G’day, I’m Tucker. My dad is David Cockerill, a software engineer here at HarperDB. I am a 3-year-old Labrador Husky mix. I love to protect my dad from all the squirrels and rabbits we have in our yard. I have very ticklish feet and love belly rubs!

4.1.0

HarperDB 4.1 introduces the ability to use worker threads for concurrently handling HTTP requests. Previously this was handled by processes. This shift provides important benefits in terms of better control of traffic delegation with support for optimized load tracking and session affinity, better debuggability, and reduced memory footprint.

This means debugging will be much easier for custom functions. If you install/run HarperDB locally, most modern IDEs like WebStorm and VSCode support worker thread debugging, so you can start HarperDB in your IDE, and set breakpoints in your custom functions and debug them.

The associated routing functionality now includes session affinity support. This can be used to consistently route users to the same thread which can improve caching locality, performance, and fairness. This can be enabled in with the http.sessionAffinity option in your configuration.

HarperDB 4.1's NoSQL query handling has been revamped to consistently use iterators, which provide an extremely memory efficient mechanism for directly streaming query results to the network as the query results are computed. This results in faster Time to First Byte (TTFB) (only the first record/value in a query needs to be computed before data can start to be sent), and less memory usage during querying (the entire query result does not need to be stored in memory). These iterators are also available in query results for custom functions and can provide means for custom function code to iteratively access data from the database without loading entire results. This should be a completely transparent upgrade, all HTTP APIs function the same, with the one exception that custom functions need to be aware that they can't access query results by [index] (they should use array methods or for-in loops to handle query results).

4.1 includes configuration options for specifying the location of database storage files. This allows you to specifically locate database directories and files on different volumes for better flexibility and utilization of disks and storage volumes. See the storage configuration and schemas configuration for information on how to configure these locations.

Logging has been revamped and condensed into one hdb.log file. See logging for more information.

A new operation called cluster_network was added, this operation will ping the cluster and return a list of enmeshed nodes.

Custom Functions will no longer automatically load static file routes, instead the @fastify/static plugin will need to be registered with the Custom Function server. See Host A Static Web UI-static.

Updates to S3 import and export mean that these operations now require the bucket region in the request. Also, if referencing a nested object it should be done in the key parameter. See examples here.

Due to the AWS SDK v2 reaching end of life support we have updated to v3. This has caused some breaking changes in our operations import_from_s3 and export_to_s3:

A new attribute region will need to be supplied
The bucket attribute can no longer have trailing slashes. Slashes will now need to be in the key.

Starting HarperDB without any command (just harperdb) now runs HarperDB like a standard process, in the foreground. This means you can use standard unix tooling for interacting with the process and is conducive for running HarperDB with systemd or any other process management tool. If you wish to have HarperDB launch itself in separate background process (and immediately terminate the shell process), you can do so by running harperdb start.

Internal Tickets completed:

CORE-609 - Ensure that attribute names are always added to global schema as Strings
CORE-1549 - Remove fastify-static code from Custom Functions server which auto serves content from "static" folder
CORE-1655 - Iterator based queries
CORE-1764 - Fix issue where describe_all operation returns an empty object for non super-users if schema(s) do not yet have table(s)
CORE-1854 - Switch to using worker threads instead of processes for handling concurrency
CORE-1877 - Extend the csv_url_load operation to allow for additional headers to be passed to the remote server when the csv is being downloaded
CORE-1893 - Add last updated timestamp to describe operations
CORE-1896 - Fix issue where Select * from system.hdb_info returns wrong HDB version number after Instance Upgrade
CORE-1904 - Fix issue when executing GEOJSON query in SQL
CORE-1905 - Add HarperDB YAML configuration setting which defines the storage location of NATS streams
CORE-1906 - Add HarperDB YAML configuration setting defining the storage location of tables.
CORE-1655 - Streaming binary format serialization
CORE-1943 - Add configuration option to set mount point for audit tables
CORE-1921 - Update NATS transaction lifecycle to handle message deduplication in work queue streams.
CORE-1963 - Update logging for better readability, reduced duplication, and request context information.
CORE-1968 - In server\nats\natsIngestService.js remove the js_msg.working(); line to improve performance.
CORE-1976 - Fix error when calling describe_table operation with no schema or table defined in payload.
CORE-1983 - Fix issue where create_attribute operation does not validate request for required attributes
CORE-2015 - Remove PM2 logs that get logged in console when starting HDB
CORE-2048 - systemd script for 4.1
CORE-2052 - Include thread information in system_information for visibility of threads
CORE-2061 - Add a better error msg when clustering is enabled without a cluster user set
CORE-2068 - Create new log rotate logic since pm2 log-rotate no longer used
CORE-2072 - Update to Node 18.15.0
CORE-2090 - Upgrade Testing from v4.0.x and v3.x to v4.1.
CORE-2091 - Run the performance tests
CORE-2092 - Allow for automatic patch version updates of certain packages
CORE-2109 - Add verify option to clustering TLS configuration
CORE-2111 - Update AWS SDK to v3

4.0.6

03/09/2023

Bug Fixes

Fixed a data serialization error that occurs when a large number of different record structures are persisted in a single table.

4.0.5

02/15/2023

Bug Fixes

CORE-2029 Improved the upgrade process for handling existing user TLS certificates and correctly configuring TLS settings. Added a prompt to upgrade to determine if new certificates should be created or existing certificates should be kept/used.
Fix the way NATS connections are honored in a local environment.
Do not define the certificate authority path to NATS if it is not defined in the HarperDB config.

4.0.4

01/27/2023

Bug Fixes

CORE-2009 Fixed bug where add node was not being called when upgrading clustering.

4.0.3

01/26/2023

Bug Fixes

CORE-2007 Add update nodes 4.0.0 launch script to build script to fix clustering upgrade.

4.0.2

01/24/2023

Bug Fixes

CORE-2003 Fix bug where if machine had one core thread config would default to zero.
Update to lmdb 2.7.3 and msgpackr 1.7.0

4.0.1

01/20/2023

Bug Fixes

CORE-1992 Local studio was not loading because the path got mangled in the build.
CORE-2001 Fixed deploy_custom_function_project after node update broke it.

4.0.0

11/2/2022

Networking & Data Replication (Clustering)

The HarperDB clustering internals have been rewritten and the underlying technology for Clustering has been completely replaced with NATS, an enterprise grade connective technology responsible for addressing, discovery and exchanging of messages that drive the common patterns in distributed systems.

CORE-1464, CORE-1470, : Remove SocketCluster dependencies and all code related to them.
CORE-1465, CORE-1485, CORE-1537, CORE-1538, CORE-1558, CORE-1583, CORE_1665, CORE-1710, CORE-1801, CORE-1865 :Add nats-server code as dependency, on install of HarperDB download nats-server is possible else fallback to building from source code.
CORE-1593, CORE-1761: Add nats.js as project dependency.
CORE-1466: Build NATS configs on harperdb run based on HarperDB YAML configuration.
CORE-1467, CORE-1508: Launch and manage NATS servers with PM2.
CORE-1468, CORE-1507: Create a process which reads the work queue stream and processes transactions.
CORE-1481, CORE-1529, CORE-1698, CORE-1502, CORE-1696: On upgrade to 4.0, update pre-existing clustering configurations, create table transaction streams, create work queue stream, update hdb_nodes table, create clustering folder structure, and rebuild self-signed certs.
CORE-1494, CORE-1521, CORE-1755: Build out internals to interface with NATS.
CORE-1504: Update existing hooks to save transactions to work with NATS.
CORE-1514, CORE-1515, CORE-1516, CORE-1527, CORE-1532: Update add_node, update_node, and remove_node operations to no longer need host and port in payload. These operations now manage dynamically sourcing of table level transaction streams between nodes and work queues.
CORE-1522: Create NATSReplyService process which handles the receiving NATS based requests from remote instances and sending back appropriate responses.
CORE-1471, CORE-1568, CORE-1563, CORE-1534, CORE-1569: Update cluster_status operation.
CORE-1611: Update pre-existing transaction log operations to be audit log operations.
CORE-1541, CORE-1612, CORE-1613: Create translation log operations which interface with streams.
CORE-1668: Update NATS serialization / deserialization to use MessagePack.
CORE-1673: Add system_info param to hdb_nodes table and update on add_node and cluster_status.
CORE-1477, CORE-1493, CORE-1557, CORE-1596, CORE-1577: Both a full HarperDB restart & just clustering restart call the NATS server with a reload directive to maintain full uptime while servers refresh.
CORE-1474:HarperDB install adds clustering folder structure.
CORE-1530: Post drop_table HarperDB purges the related transaction stream.
CORE-1567: Set NATS config to always use TLS.
CORE-1543: Removed the transact_to_cluster attribute from the bulk load operations. Now bulk loads always replicate.
CORE-1533, CORE-1556, CORE-1561, CORE-1562, CORE-1564: New operation configure_cluster, this operation enables bulk publishing and subscription of multiple tables to multiple instances of HarperDB.
CORE-1535: Create work queue stream on install of HarperDB. This stream receives transactions from remote instances of HarperDB which are then ingested in order.
CORE-1551: Create transaction streams on the remote node if they do not exist when performing add_node or update_node.
CORE-1594, CORE-1605, CORE-1749, CORE-1767, CORE-1770: Optimize the work queue stream and its consumer to be more performant and validate exact once delivery.
CORE-1621, CORE-1692, CORE-1570, CORE-1693: NATS stream names are MD5 hashed to avoid characters that HarperDB allows, but NATS may not.
CORE-1762: Add a new optional attribute to add_node and update_node named opt_start_time. This attribute sets a starting time to start synchronizing transactions.
CORE-1785: Optimizations and bug fixes in regards to sourcing data from remote instances on HarperDB.
CORE-1588: Created new operation set_cluster_routes to enable setting routes for instances of HarperDB to mesh together.
CORE-1589: Created new operation get_cluster_routes to allow for retrieval of routes used to connect the instance of HarperDB to the mesh.
CORE-1590: Created new operation delete_cluster_routes to allow for removal of routes used to connect the instance of HarperDB to the mesh.
CORE-1667: Fix old environment variable CLUSTERING_PORT not mapping to new hub server port.
CORE-1609: Allow remove_node to be called when the other node cannot be reached.
CORE-1815: Add transaction lock to add_node and update_node to avoid concurrent nats source update bug.
CORE-1848: Update stream configs if the node name has been changed in the YAML configuration.
CORE-1873: Update add_node and update_node so that it auto-creates schema/table on both local and remote node respectively

Data Storage

We have made improvements to how we store, index, and retrieve data.

CORE-1619: Enabled new concurrent flushing technology for improved write performance.
CORE-1701: Optimize search performance for search_by_conditions when executing multiple AND conditions.
CORE-1652: Encode the values of secondary indices more efficiently for faster access.
CORE-1670: Store updated timestamp in lmdb.js' version property.
CORE-1651: Enabled multiple value indexing of array values which allows for the ability to search on specific elements in an array more efficiently.
CORE-1649, CORE-1659: Large text values (larger than 255 bytes) are no longer stored in separate blob index. Now they are segmented and delimited in the same index to increase search performance.
Complex objects and object arrays are no longer stored in a separate index to preserve storage and increase write throughput.
CORE-1650, CORE-1724, CORE-1738: Improved internals around interpreting attribute values.
CORE-1657: Deferred property decoding allows large objects to be stored, but individual attributes can be accessed (like with get_attributes) without incurring the cost of decoding the entire object.
CORE-1658: Enable in-memory caching of records for even faster access to frequently accessed data.
CORE-1693: Wrap updates in async transactions to ensure ACID-compliant updates.
CORE-1653: Upgrade to 4.0 rebuilds tables to reflect changes made to index improvements.
CORE-1753: Removed old node-lmdb dependency.
CORE-1787: Freeze objects returned from queries.
CORE-1821: Read the WRITE_ASYNC setting which enables LMDB nosync.

Logging

HarperDB has increased logging specificity by breaking out logs based on components logging. There are specific log files each for HarperDB Core, Custom Functions, Hub Server, Leaf Server, and more.

CORE-1497: Remove pino and winston dependencies.
CORE-1426: All logging is output via stdout and stderr, our default logging is then picked up by PM2 which handles writing out to file.
CORE-1431: Improved read_log operation validation.
CORE-1433, CORE-1463: Added log rotation.
CORE-1553, CORE-1555, CORE-1552, CORE-1554, CORE-1704: Performance gain by only serializing objects and arrays if the log is for the level defined in configuration.
CORE-1436: Upgrade to 4.0 updates internals for logging changes.
CORE-1428, CORE-1440, CORE-1442, CORE-1434, CORE-1435, CORE-1439, CORE-1482, CORE-1751, CORE-1752: Bug fixes, performance improvements and improved unit tests.
CORE-1691: Convert non-PM2 managed log file writes to use Node.js fs.appendFileSync function.

Configuration

HarperDB has updated its configuration from a properties file to YAML.

CORE-1448, CORE-1449, CORE-1519, CORE-1587: Upgrade automatically converts the pre-existing settings file to YAML.
CORE-1445, CORE-1534, CORE-1444, CORE-1858: Build out new logic to create, update, and interpret the YAML configuration file.
Installer has updated prompts to reflect YAML settings.
CORE-1447: Create an alias for the configure_cluster operation as set_configuration.
CORE-1461, CORE-1462, CORE-1483: Unit test improvements.
CORE-1492: Improvements to get_configuration and set_configuration operations.
CORE-1503: Modify HarperDB configuration for more granular certificate definition.
CORE-1591: Update routes IP param to host and to leaf config in harperdb.conf
CORE-1519: Fix issue when switching between old and new versions of HarperDB we are getting the config parameter is undefined error on npm install.

Broad NodeJS and Platform Support

CORE-1624: HarperDB can now run on multiple versions of NodeJS, from v14 to v19. We primarily test on v18, so that is the preferred version.

Windows 10 and 11

CORE-1088: HarperDB now runs natively on Windows 10 and 11 without the need to run in a container or installed in WSL. Windows is only intended for evaluation and development purposes, not for production work loads.

Extra Changes and Bug Fixes

CORE-1520: Refactor installer to remove all waterfall code and update to use Promises.
CORE-1573: Stop the PM2 daemon and any logging processes when stopping hdb.
CORE-1586: When HarperDB is running in foreground stop any additional logging processes from being spawned.
CORE-1626: Update docker file to accommodate new harperdb.conf file.
CORE-1592, CORE-1526, CORE-1660, CORE-1646, CORE-1640, CORE-1689, CORE-1711, CORE-1601, CORE-1726, CORE-1728, CORE-1736, CORE-1735, CORE-1745, CORE-1729, CORE-1748, CORE-1644, CORE-1750, CORE-1757, CORE-1727, CORE-1740, CORE-1730, CORE-1777, CORE-1778, CORE-1782, CORE-1775, CORE-1771, CORE-1774, CORE-1759, CORE-1772, CORE-1861, CORE-1862, CORE-1863, CORE-1870, CORE-1869:Changes for CI/CD pipeline and integration tests.
CORE-1661: Fixed issue where old boot properties file caused an error when attempting to install 4.0.0.
CORE-1697, CORE-1814, CORE-1855: Upgrade fastify dependency to new major version 4.
CORE-1629: Jobs are now running as processes managed by the PM2 daemon.
CORE-1733: Update LICENSE to reflect our EULA on our site.
CORE-1606: Enable Custom Functions by default.
CORE-1714: Include pre-built binaries for most common platforms (darwin-arm64, darwin-x64, linux-arm64, linux-x64, win32-x64).
CORE-1628: Fix issue where setting license through environment variable not working.
CORE-1602, CORE-1760, CORE-1838, CORE-1839, CORE-1847, CORE-1773: HarperDB Docker container improvements.
CORE-1706: Add support for encoding HTTP responses with MessagePack.
CORE-1709: Improve the way lmdb.js dependencies are installed.
CORE-1758: Remove/update unnecessary HTTP headers.
CORE-1756: On npm install and harperdb install change the node version check from an error to a warning if the installed Node.js version does not match our preferred version.
CORE-1791: Optimizations to authenticated user caching.
CORE-1794: Update README to discuss Windows support & Node.js versions
CORE-1837: Fix issue where Custom Function directory was not being created on install.
CORE-1742: Add more validation to audit log - check schema/table exists and log is enabled.
CORE-1768: Fix issue where when running in foreground HarperDB process is not stopping on harperdb stop.
CORE-1864: Fix to semver checks on upgrade.
CORE-1850: Fix issue where a cluster_user type role could not be altered.

HarperDB Monkey (Version 3)

Did you know our release names are dedicated to employee pups? For our third release, we have Monkey.

Hi, I am Monkey, a.k.a. Monk, a.k.a. Monchichi. My dad is Aron Johnson, the Director of DevOps at HarperDB. I am an eight-year-old Australian Cattle dog mutt whose favorite pastime is hunting and collecting tennis balls from the park next to her home. I love burrowing in the Colorado snow, rolling in the cool grass on warm days, and cheese!

3.3.0

CORE-1595 Added new role type structure_user, this enables non-superusers to be able to create/drop schema/table/attribute.
CORE-1501 Improved performance for drop_table.
CORE-1599 Added two new operations for custom functions install_node_modules & audit_node_modules.
CORE-1598 Added skip_node_modules flag to package_custom_function_project operation. This flag allows for not bundling project dependencies and deploying a smaller project to other nodes. Use this flag in tandem with install_node_modules.
CORE-1707 Binaries are now included for Linux on AMD64, Linux on ARM64, and macOS. GCC, Make, Python are no longer required when installing on these platforms.

3.2.1

6/1/2022

Features/Updates

CORE-1573 Added logic to track the pid of the foreground process if running in foreground. Then on stop, use that pid to kill the process. Logic was also added to kill the pm2 daemon when stop is called.

3.2.0

3/25/2022

Features/Updates

CORE-1391 Bug fix related to orphaned HarperDB background processes.
CORE-1509 Updated node version check, updated Node.js version, updated project dependencies.
CORE-1518 Remove final call from logger.

3.1.5

3/4/2022

Features/Updates

CORE-1498 Fixed incorrect autocasting of string that start with "0." that tries to convert to number but instead returns NaN.

3.1.4

2/24/2022

Features/Updates

CORE-1460 Added new setting STORAGE_WRITE_ASYNC. If this setting is true, LMDB will have faster write performance at the expense of not being crash safe. The default for this setting is false, which results in HarperDB being crash safe.

3.1.3

1/14/2022

Bug Fixes

CORE-1446 Fix for scans on indexes larger than 1 million entries causing queries to never return

3.1.2

10/21/2021

Features/Updates

Updated the installation ASCII art to reflect the new HarperDB logo

Bug Fixes

CORE-1408 Corrects issue where drop_attribute was not properly setting the LMDB version number causing tables to behave unexpectedly

3.1.1

9/23/2021

Features/Updates

CORE-1393 Added utility function to add settings from env/cmd vars to the settings file on every run/restart
CORE-1395 Create a setting which will allow to enable the local Studio to be served from an instance of HarperDB
CORE-1397 Update the stock 404 response to not return the request URL
General updates to optimize Docker container

Bug Fixes

CORE-1399 Added fixes for complex SQL alias issues

3.1.0

8/24/2021

Features/Updates

CORE-1320, CORE-1321, CORE-1323, CORE-1324 Version 1.0 of HarperDB Custom Functions
CORE-1275, CORE-1276, CORE-1278, CORE-1279, CORE-1280, CORE-1282, CORE-1283, CORE-1305, CORE-1314 IPC server for communication between HarperDB processes, including HarperDB, HarperDB Clustering, and HarperDB Functions
CORE-1352, CORE-1355, CORE-1356, CORE-1358 Implement pm2 for HarperDB process management
CORE-1292, CORE-1308, CORE-1312, CORE-1334, CORE-1338 Updated installation process to start HarperDB immediately on install and to accept all config settings via environment variable or command line arguments
CORE-1310 Updated licensing functionality
CORE-1301 Updated validation for performance improvement
CORE-1359 Add hdb-response-time header which returns the HarperDB response time in milliseconds
CORE-1330, CORE-1309 New config settings: LOG_TO_FILE, LOG_TO_STDSTREAMS, IPC_SERVER_PORT, RUN_IN_FOREGROUND, CUSTOM_FUNCTIONS, CUSTOM_FUNCTIONS_PORT, CUSTOM_FUNCTIONS_DIRECTORY, MAX_CUSTOM_FUNCTION_PROCESSES

Bug Fixes

CORE-1315 Corrected issue in HarperDB restart scenario
CORE-1370 Update some of the validation error handlers so that they don't log full stack

3.0.0

5/18/2021

Features/Updates

CORE-1217, CORE-1226, CORE-1232 Create new search_by_conditions operation.
CORE-1304 Upgrade to Node 12.22.1.
CORE-1235 Adds new upgrade/install functionality.
CORE-1206, CORE-1248, CORE-1252 Implement lmdb-store library for optimized performance.
CORE-1062 Added alias operation for delete_files_before, named delete_records_before.
CORE-1243 Change HTTPS_ON settings value to false by default.
CORE-1189 Implement fastify web server, resulting in improved performance.
CORE-1221 Update user API to use role name instead of role id.
CORE-1225 Updated dependencies to eliminate npm security warnings.
CORE-1241 Adds 3.0 update directive and refactors/fixes update functionality.

Bug Fixes

CORE-1299 Remove all references to the PROJECT_DIR setting. This setting is problematic when using node version managers and upgrading the version of node and then installing a new instance of HarperDB.
CORE-1288 Fix bug with drop table/schema that was causing 'env required' error log.
CORE-1285 Update warning log when trying to create an attribute that already exists.
CORE-1254 Added logic to manage data collisions in clustering.
CORE-1212 Add pre-check to drop_user that returns error if user doesn't exist.
CORE-1114 Update response code and message from add_user when user already exists.
CORE-1111 Update response from create_attribute to match the create schema/table response.
CORE-1205 Fixed bug that prevented schema/table from being dropped if name was a number or had a wildcard value in it. Updated validation for insert, upsert and update.

HarperDB Penny (Version 2)

Did you know our release names are dedicated to employee pups? For our second release, Penny was the star.

Here is a bit about Penny:

Hi I am Penny! My dad is Kyle Bernhardy, the CTO of HarperDB. I am a nine-year-old Whippet who lives for running hard and fast while exploring the beautiful terrain of Colorado. My favorite activity is chasing birds along with afternoon snoozes in a sunny spot in my backyard.

2.3.1

1/29/2021

Bug Fixes

CORE-1218 A bug in HarperDB 2.3.0 was identified related to manually calling the create_attribute operation. This bug caused secondary indexes to be overwritten by the most recently inserted or updated value for the index, thereby causing a search operation filtered with that index to only return the most recently inserted/updated row. Note, this issue does not affect attributes that are reflexively/automatically created. It only affects attributes created using create_attribute. To resolve this issue in 2.3.0 or earlier, drop and recreate your table using reflexive attribute creation. In 2.3.1, drop and recreate your table and use either reflexive attribute creation or create_attribute.
CORE-1219 Increased maximum table attributes from 1000 to 10000

2.3.0

12/03/2020

Features/Updates

CORE-1191, CORE-1190, CORE-1125, CORE-1157, CORE-1126, CORE-1140, CORE-1134, CORE-1123, CORE-1124, CORE-1122 Added JWT Authentication option (See documentation for more information)
CORE-1128, CORE-1143, CORE-1140, CORE-1129 Added upsert operation
CORE-1187 Added get_configuration operation which allows admins to view their configuration settings.
CORE-1175 Added new internal LMDB function to copy an environment for use in future features.
CORE-1166 Updated packages to address security vulnerabilities.

Bug Fixes

CORE-1195 Modified drop_attribute to drop after data cleanse completes.
CORE-1149 Fix SQL bug regarding self joins and updates alasql to 0.6.5 release.
CORE-1168 Fix inconsistent invalid schema/table errors.
CORE-1162 Fix bug which caused delete_files_before to cause tables to grow in size due to an open cursor issue.

2.2.3

11/16/2020

CORE-1158 Performance improvements to core delete function and configuration of delete_files_before to run in batches with a pause into between.

2.2.2

10/27/2020

CORE-1154 Allowed transaction logging to be disabled even if clustering is enabled.
CORE-1153 Fixed issue where delete_files_before was writing to transaction log.
CORE-1152 Fixed issue where no more than 4 HarperDB forks would be created.
CORE-1112 Adds handling for system timestamp attributes in permissions.
CORE-1131 Adds better handling for checking perms on operations with action value in JSON.
CORE-1113 Fixes validation bug checking for super user/cluster user permissions and other permissions.
CORE-1135 Adds validation for valid keys in role API operations.
CORE-1073 Adds new import_from_s3 operation to API.

2.2.0

08/24/2020

Features/Updates

CORE-997 Updated the data format for CSV data loads being sync'd across a cluster to take up less resources
CORE-1018 Adds SQL functionality for BETWEEN statements
CORE-1032 Updates permissions to allow regular users (i.e. non-super users) to call the get_job operation
CORE-1036 On create/drop table we auto create/drop the related transactions environments for the schema.table
CORE-1042 Built raw functions to write to a tables transaction log for insert/update/delete operations
CORE-1057 Implemented write transaction into lmdb create/update/delete functions
CORE-1048 Adds SEARCH wildcard handling for role permissions standards
CORE-1059 Added config setting to disable transaction logging for an instance
CORE-1076 Adds permissions filter to describe operations
CORE-1043 Change clustering catchup to use the new transaction log
CORE-1052 Removed word "master" from source
CORE-1061 Added new operation called delete_transactions_before this will tail a transaction log for a specific schema / table
CORE-1040 On HarperDB startup make sure all tables have a transaction environment
CORE-1055 Added 2 new setting to change the server headersTimeout & keepAliveTimeout from the config file
CORE-1044 Created new operation read_transaction_log which will allow a user to get transactions for a table by timestamp, username, or hash_value
CORE-1043 Change clustering catchup to use the new transaction log
CORE-1089 Added new attribute to system_information for table/transaction log data size in bytes & transaction log record count
CORE-1101 Fix to store empty strings rather than considering them null & fix to be able to search on empty strings in SQL/NoSQL.
CORE-1054 Updates permissions object to remove delete attribute permission and update table attribute permission key to attribute_permissions
CORE-1092 Do not allow the __createdtime__ to be updated
CORE-1085 Updates create schema/table & drop schema/table/attribute operations permissions to require super user role and adds integration tests to validate
CORE-1071 Updates response messages and status codes from describe_schema and describe_table operations to provide standard language/status code when a schema item is not found
CORE-1049 Updates response message for SQL update op with no matching rows
CORE-1096 Added tracking of the origin in the transaction log. This origin object stores the node name, timestamp of the transaction from the originating node & the user.

Bug Fixes

CORE-1028 Fixes bug for simple SQL SELECT queries not returning aliases and incorrectly returning hash values when not requested in query
CORE-1037 Fixed an issue where numbers with leading zero i.e. 00123 are converted to numbers rather than being honored as strings.
CORE-1063 Updates permission error response shape to consolidate issues into individual objects per schema/table combo
CORE-1098 Fixed an issue where transaction environments were remaining in the global cache after being dropped.
CORE-1086 Fixed issue where responses from insert/update were incorrect with skipped records.
CORE-1079 Fixes SQL bugs around invalid schema/table and special characters in WHERE clause

2.1.1

05/22/2020

Highlights

CORE-1007 Added the ability to perform SQL INSERT & UPDATE with function calls & expressions on values.
CORE-1023 Fixed minor bug in final SQL step incorrectly trying to translate ordinals to alias in ORDER BY statement.
CORE-1020 Fixed bug allowing 'null' and 'undefined' string values to be passed in as valid hash values.
CORE-1006 Added SQL functionality that enables JOIN statements across different schemas.
CORE-1005 Implemented JSONata library to handle our JSON document search functionality in SQL, creating the SEARCH_JSON function.
CORE-1009 Updated schema validation to allow all printable ASCII characters to be used in schema/table/attribute names, except, forward slashes and backticks. Same rules apply now for hash attribute values.
CORE-1003 Fixed handling of ORDER BY statements with function aliases.
CORE-1004 Fixed bug related to SELECT* on JOIN queries with table columns with the same name.
CORE-996 Fixed an issue where the transact_to_cluster flag is lost for CSV URL loads, fixed an issue where new attributes created in CSV bulk load do not sync to the cluster.
CORE-994 Added new operation system_information. This operation returns info & metrics for the OS, time, memory, cpu, disk, network.
CORE-993 Added new custom date functions for AlaSQL & UTC updates.
CORE-991 Changed jobs to spawn a new process which will run the intended job without impacting a main HarperDB process.
CORE-992 HTTPS enabled by default.
CORE-990 Updated describe_table to add the record count for the table for LMDB data storage.
CORE-989 Killed the socket cluster processes prior to HarperDB processes to eliminate a false uptime.
CORE-975 Updated time values set by SQL Date Functions to be in epoch format.
CORE-974 Added date functions to SQL SELECT column alias functionality.

HarperDB Alby (Version 1)

Did you know our release names are dedicated to employee pups? For our first release, Alby was our pup.

Here is a bit about Alby:

Hi, I am Alby. My mom is Kaylan Stock, Director of Marketing at HarperDB. I am a 9-year-old Great Dane mix who loves sun bathing, going for swims, and wreaking havoc on the local squirrels. My favorite snack is whatever you are eating, and I love a good butt scratch!

1.3.1

2/26/2019

Features

Clustering connection direction appointment
Foundations for threading/multi processing
UUID autogen for hash attributes that were not provided
Added cluster status operation

Bug Fixes and Enhancements

More logging
Clustering communication enhancements
Clustering queue ordering by timestamps
Cluster re connection enhancements
Number of system core(s) detection
Node LTS (10.15) compatibility
Update/Alter users enhancements
General performance enhancements
Warning is logged if different versions of harperdb are connected via clustering
Fixed need to restart after user creation/alteration
Fixed SQL error that occurred on selecting from an empty table

1.3.0

11/2/2018

Features

Upgrade: Upgrade to newest version via command line.
SQL Support: Added IS NULL for SQL parser.
Added attribute validation to search operations.

Fixes

Fixed SELECT calculations, i.e. SELECT 2+2.
Fixed select OR not returning expected results.
No longer allowing reserved words for schema and table names.
Corrected process interruptions from improper SQL statements.
Improved message handling between spawned processes that replace killed processes.
Enhanced error handling for updates to tables that do not exist.
Fixed error handling for NoSQL responses when get_attributes is provided with invalid attributes.
Fixed issue with new columns not being updated properly in update statements.
Now validating roles, tables and attributes when creating or updating roles.
Fixed an issue where in some cases undefined was being returned after dropping a role

1.2.0

7/10/2018

Features

Time to Live: Conserve the resources of your edge device by setting data on devices to live for a specific period of time.
Geo: HarperDB has implemented turf.js into its SQL parser to enable geo based analytics.
Jobs: CSV Data loads, Exports & Time to Live now all run as back ground jobs.
Exports: Perform queries that export into JSON or CSV and save to disk or S3.

Fixes

Fixed issue where CSV data loads incorrectly report number of records loaded.
Added validation to stop BETWEEN operations in SQL.
Updated logging to not include internal variables in the logs.
Cleaned up add_role response to not include internal variables.
Removed old and unused dependencies.
Build out further unit tests and integration tests.
Fixed https to handle certificates properly.
Improved stability of clustering & replication.
Corrected issue where Objects and Arrays were not casting properly in SQL SELECT response.
Fixed issue where Blob text was not being returned from SQL SELECTs.
Fixed error being returned when querying on table with no data, now correctly returns empty array.
Improved performance in SQL when searching on exact values.
Fixed error when ./harperdb stop is called.
Fixed logging issue causing instability in installer.
Fixed read_log operation to accept date time.
Added permissions checking to export_to_s3.
Added ability to run SQL on SELECT without a FROM.
Fixed issue where updating a user’s password was not encrypting properly.
Fixed user_guide.html to point to readme on git repo.
Created option to have HarperDB run as a foreground process.
Updated user_info to return the correct role for a user.
Fixed issue where HarperDB would not stop if the database root was deleted.
Corrected error message on insert if an invalid schema is provided.
Added permissions checks for user & role operations.

1.1.0

4/18/2018

Features

Users & Roles:
- Limit/Assign access to all HarperDB operations
- Limit/Assign access to schemas, tables & attributes
- Limit/Assign access to specific SQL operations (INSERT, UPDATE, DELETE, SELECT)
Enhanced SQL parser
- Added extensive ANSI SQL Support.
- Added Array function, which allows for converting relational data into Object/Hierarchical data
- Distinct_Array Function: allows for removing duplicates in the Array function.
- Enhanced SQL Validation: Improved validation around structure of SQL, validating the schema, etc..
- 10x performance improvement on SQL statements.
Export Function: can now call a NoSQL/SQL search and have it export to CSV or JSON.
Added upgrade function to CLI
Added ability to perform bulk update from CSV
Created landing page for HarperDB.
Added CORS support to HarperDB

Fixes

Fixed memory leak in CSV bulk loads
Corrected error when attempting to perform a SQL DELETE
Added further validation to NoSQL UPDATE to validate schema & table exist
Fixed install issue occurring when part of the install path does not exist, the install would silently fail.
Fixed issues with replicated data when one of the replicas is down
Removed logging of initial user’s credentials during install
Can now use reserved words as aliases in SQL
Removed user(s) password in results when calling list_users
Corrected forwarding of operations to other nodes in a cluster
Corrected lag in schema meta-data passing to other nodes in a cluster
Drop table & schema now move the table & schema or table to the trash folder under the Database folder for later permanent deletion.
Bulk inserts no longer halt the entire operation if n records already exist, instead the return includes the hashes of records that have been skipped.
Added ability to accept EULA from command line
Corrected search_by_value not searching on the correct attribute
Added ability to increase the timeout of a request by adding SERVER_TIMEOUT_MS to config/settings.js
Add error handling resulting from SQL calculations.
Standardized error responses as JSON.
Corrected internal process generation to not allow more processes than machine has cores.

4.0.0

11/2/2022

Networking & Data Replication (Clustering)

CORE-1464, CORE-1470, : Remove SocketCluster dependencies and all code related to them.
CORE-1465, CORE-1485, CORE-1537, CORE-1538, CORE-1558, CORE-1583, CORE_1665, CORE-1710, CORE-1801, CORE-1865 :Add nats-server code as dependency, on install of HarperDB download nats-server is possible else fallback to building from source code.
CORE-1593, CORE-1761: Add nats.js as project dependency.
CORE-1466: Build NATS configs on harperdb run based on HarperDB YAML configuration.
CORE-1467, CORE-1508: Launch and manage NATS servers with PM2.
CORE-1468, CORE-1507: Create a process which reads the work queue stream and processes transactions.
CORE-1481, CORE-1529, CORE-1698, CORE-1502, CORE-1696: On upgrade to 4.0, update pre-existing clustering configurations, create table transaction streams, create work queue stream, update hdb_nodes table, create clustering folder structure, and rebuild self-signed certs.
CORE-1494, CORE-1521, CORE-1755: Build out internals to interface with NATS.
CORE-1504: Update existing hooks to save transactions to work with NATS.
CORE-1514, CORE-1515, CORE-1516, CORE-1527, CORE-1532: Update add_node, update_node, and remove_node operations to no longer need host and port in payload. These operations now manage dynamically sourcing of table level transaction streams between nodes and work queues.
CORE-1522: Create NATSReplyService process which handles the receiving NATS based requests from remote instances and sending back appropriate responses.
CORE-1471, CORE-1568, CORE-1563, CORE-1534, CORE-1569: Update cluster_status operation.
CORE-1611: Update pre-existing transaction log operations to be audit log operations.
CORE-1541, CORE-1612, CORE-1613: Create translation log operations which interface with streams.
CORE-1668: Update NATS serialization / deserialization to use MessagePack.
CORE-1673: Add system_info param to hdb_nodes table and update on add_node and cluster_status.
CORE-1477, CORE-1493, CORE-1557, CORE-1596, CORE-1577: Both a full HarperDB restart & just clustering restart call the NATS server with a reload directive to maintain full uptime while servers refresh.
CORE-1474:HarperDB install adds clustering folder structure.
CORE-1530: Post drop_table HarperDB purges the related transaction stream.
CORE-1567: Set NATS config to always use TLS.
CORE-1543: Removed the transact_to_cluster attribute from the bulk load operations. Now bulk loads always replicate.
CORE-1533, CORE-1556, CORE-1561, CORE-1562, CORE-1564: New operation configure_cluster, this operation enables bulk publishing and subscription of multiple tables to multiple instances of HarperDB.
CORE-1535: Create work queue stream on install of HarperDB. This stream receives transactions from remote instances of HarperDB which are then ingested in order.
CORE-1551: Create transaction streams on the remote node if they do not exist when performing add_node or update_node.
CORE-1594, CORE-1605, CORE-1749, CORE-1767, CORE-1770: Optimize the work queue stream and its consumer to be more performant and validate exact once delivery.
CORE-1621, CORE-1692, CORE-1570, CORE-1693: NATS stream names are MD5 hashed to avoid characters that HarperDB allows, but NATS may not.
CORE-1762: Add a new optional attribute to add_node and update_node named opt_start_time. This attribute sets a starting time to start synchronizing transactions.
CORE-1785: Optimizations and bug fixes in regards to sourcing data from remote instances on HarperDB.
CORE-1588: Created new operation set_cluster_routes to enable setting routes for instances of HarperDB to mesh together.
CORE-1589: Created new operation get_cluster_routes to allow for retrieval of routes used to connect the instance of HarperDB to the mesh.
CORE-1590: Created new operation delete_cluster_routes to allow for removal of routes used to connect the instance of HarperDB to the mesh.
CORE-1667: Fix old environment variable CLUSTERING_PORT not mapping to new hub server port.
CORE-1609: Allow remove_node to be called when the other node cannot be reached.
CORE-1815: Add transaction lock to add_node and update_node to avoid concurrent nats source update bug.
CORE-1848: Update stream configs if the node name has been changed in the YAML configuration.
CORE-1873: Update add_node and update_node so that it auto-creates schema/table on both local and remote node respectively

Data Storage

We have made improvements to how we store, index, and retrieve data.

CORE-1619: Enabled new concurrent flushing technology for improved write performance.
CORE-1701: Optimize search performance for search_by_conditions when executing multiple AND conditions.
CORE-1652: Encode the values of secondary indices more efficiently for faster access.
CORE-1670: Store updated timestamp in lmdb.js' version property.
CORE-1651: Enabled multiple value indexing of array values which allows for the ability to search on specific elements in an array more efficiently.
CORE-1649, CORE-1659: Large text values (larger than 255 bytes) are no longer stored in separate blob index. Now they are segmented and delimited in the same index to increase search performance.
Complex objects and object arrays are no longer stored in a separate index to preserve storage and increase write throughput.
CORE-1650, CORE-1724, CORE-1738: Improved internals around interpreting attribute values.
CORE-1657: Deferred property decoding allows large objects to be stored, but individual attributes can be accessed (like with get_attributes) without incurring the cost of decoding the entire object.
CORE-1658: Enable in-memory caching of records for even faster access to frequently accessed data.
CORE-1693: Wrap updates in async transactions to ensure ACID-compliant updates.
CORE-1653: Upgrade to 4.0 rebuilds tables to reflect changes made to index improvements.
CORE-1753: Removed old node-lmdb dependency.
CORE-1787: Freeze objects returned from queries.
CORE-1821: Read the WRITE_ASYNC setting which enables LMDB nosync.

Logging

HarperDB has increased logging specificity by breaking out logs based on components logging. There are specific log files each for HarperDB Core, Custom Functions, Hub Server, Leaf Server, and more.

CORE-1497: Remove pino and winston dependencies.
CORE-1426: All logging is output via stdout and stderr, our default logging is then picked up by PM2 which handles writing out to file.
CORE-1431: Improved read_log operation validation.
CORE-1433, CORE-1463: Added log rotation.
CORE-1553, CORE-1555, CORE-1552, CORE-1554, CORE-1704: Performance gain by only serializing objects and arrays if the log is for the level defined in configuration.
CORE-1436: Upgrade to 4.0 updates internals for logging changes.
CORE-1428, CORE-1440, CORE-1442, CORE-1434, CORE-1435, CORE-1439, CORE-1482, CORE-1751, CORE-1752: Bug fixes, performance improvements and improved unit tests.
CORE-1691: Convert non-PM2 managed log file writes to use Node.js fs.appendFileSync function.

Configuration

HarperDB has updated its configuration from a properties file to YAML.

CORE-1448, CORE-1449, CORE-1519, CORE-1587: Upgrade automatically converts the pre-existing settings file to YAML.
CORE-1445, CORE-1534, CORE-1444, CORE-1858: Build out new logic to create, update, and interpret the YAML configuration file.
Installer has updated prompts to reflect YAML settings.
CORE-1447: Create an alias for the configure_cluster operation as set_configuration.
CORE-1461, CORE-1462, CORE-1483: Unit test improvements.
CORE-1492: Improvements to get_configuration and set_configuration operations.
CORE-1503: Modify HarperDB configuration for more granular certificate definition.
CORE-1591: Update routes IP param to host and to leaf config in harperdb.conf
CORE-1519: Fix issue when switching between old and new versions of HarperDB we are getting the config parameter is undefined error on npm install.

Broad NodeJS and Platform Support

CORE-1624: HarperDB can now run on multiple versions of NodeJS, from v14 to v19. We primarily test on v18, so that is the preferred version.

Windows 10 and 11

CORE-1088: HarperDB now runs natively on Windows 10 and 11 without the need to run in a container or installed in WSL. Windows is only intended for evaluation and development purposes, not for production work loads.

Extra Changes and Bug Fixes

CORE-1520: Refactor installer to remove all waterfall code and update to use Promises.
CORE-1573: Stop the PM2 daemon and any logging processes when stopping hdb.
CORE-1586: When HarperDB is running in foreground stop any additional logging processes from being spawned.
CORE-1626: Update docker file to accommodate new harperdb.conf file.
CORE-1592, CORE-1526, CORE-1660, CORE-1646, CORE-1640, CORE-1689, CORE-1711, CORE-1601, CORE-1726, CORE-1728, CORE-1736, CORE-1735, CORE-1745, CORE-1729, CORE-1748, CORE-1644, CORE-1750, CORE-1757, CORE-1727, CORE-1740, CORE-1730, CORE-1777, CORE-1778, CORE-1782, CORE-1775, CORE-1771, CORE-1774, CORE-1759, CORE-1772, CORE-1861, CORE-1862, CORE-1863, CORE-1870, CORE-1869:Changes for CI/CD pipeline and integration tests.
CORE-1661: Fixed issue where old boot properties file caused an error when attempting to install 4.0.0.
CORE-1697, CORE-1814, CORE-1855: Upgrade fastify dependency to new major version 4.
CORE-1629: Jobs are now running as processes managed by the PM2 daemon.
CORE-1733: Update LICENSE to reflect our EULA on our site.
CORE-1606: Enable Custom Functions by default.
CORE-1714: Include pre-built binaries for most common platforms (darwin-arm64, darwin-x64, linux-arm64, linux-x64, win32-x64).
CORE-1628: Fix issue where setting license through environment variable not working.
CORE-1602, CORE-1760, CORE-1838, CORE-1839, CORE-1847, CORE-1773: HarperDB Docker container improvements.
CORE-1706: Add support for encoding HTTP responses with MessagePack.
CORE-1709: Improve the way lmdb.js dependencies are installed.
CORE-1758: Remove/update unnecessary HTTP headers.
CORE-1756: On npm install and harperdb install change the node version check from an error to a warning if the installed Node.js version does not match our preferred version.
CORE-1791: Optimizations to authenticated user caching.
CORE-1794: Update README to discuss Windows support & Node.js versions
CORE-1837: Fix issue where Custom Function directory was not being created on install.
CORE-1742: Add more validation to audit log - check schema/table exists and log is enabled.
CORE-1768: Fix issue where when running in foreground HarperDB process is not stopping on harperdb stop.
CORE-1864: Fix to semver checks on upgrade.
CORE-1850: Fix issue where a cluster_user type role could not be altered.