HarperDB’s Custom Functions is built on top of Fastify, so our route definitions follow their specifications. Below is a very simple example of a route declaration.

Route URLs are resolved in the following manner:

• [Instance URL]:[Custom Functions Port]/[Project Name]/[Route URL]

• The route below, within the dogs project, with a route of breeds would be available at http://localhost:9926/dogs/breeds.

In effect, this route is just a pass-through to HarperDB. The same result could have been achieved by hitting the core HarperDB API, since it uses hdbCore.preValidation and hdbCore.request, which are defined in the “helper methods” section, below.

module.exports = async (server, { hdbCore, logger }) => {
    server.route({
        url: '/',
        method: 'POST',
        preValidation: hdbCore.preValidation,
        handler: hdbCore.request,
    })
}

Custom Handlers

For endpoints where you want to execute multiple operations against HarperDB, or perform additional processing (like an ML classification, or an aggregation, or a call to a 3rd party API), you can define your own logic in the handler. The function below will execute a query against the dogs table, and filter the results to only return those dogs over 4 years in age.

IMPORTANT: This route has NO preValidation and uses hdbCore.requestWithoutAuthentication, which- as the name implies- bypasses all user authentication. See the security concerns and mitigations in the “helper methods” section, below.

module.exports = async (server, { hdbCore, logger }) => {
    server.route({
        url: '/:id',
        method: 'GET',
        handler: (request) => {
            request.body= {
                operation: 'sql',
                sql: `SELECT * FROM dev.dog WHERE id = ${request.params.id}`
            };

          const result = await hdbCore.requestWithoutAuthentication(request);
          return result.filter((dog) => dog.age > 4);
        }
    });
}

Custom preValidation Hooks

The simple example above was just a pass-through to HarperDB- the exact same result could have been achieved by hitting the core HarperDB API. But for many applications, you may want to authenticate the user using custom logic you write, or by conferring with a 3rd party service. Custom preValidation hooks let you do just that.

Below is an example of a route that uses a custom validation hook:

const customValidation = require('../helpers/customValidation');

module.exports = async (server, { hdbCore, logger }) => {
    server.route({
        url: '/:id',
        method: 'GET',
        preValidation: (request) => customValidation(request, logger),
        handler: (request) => {
            request.body= {
                operation: 'sql',
                sql: `SELECT * FROM dev.dog WHERE id = ${request.params.id}`
            };

         return hdbCore.requestWithoutAuthentication(request);
        }
    });
}

Notice we imported customValidation from the helpers directory. To include a helper, and to see the actual code within customValidation, see Define Helpers.

Helper Methods

When declaring routes, you are given access to 2 helper methods: hdbCore and logger.

hdbCore

hdbCore contains three functions that allow you to authenticate an inbound request, and execute operations against HarperDB directly, by passing the standard Operations API.

• preValidation

  This takes the authorization header from the inbound request and executes the same authentication as the standard HarperDB Operations API. It will determine if the user exists, and if they are allowed to perform this operation. If you use the request method, you have to use preValidation to get the authenticated user.

• request

  This will execute a request with HarperDB using the operations API. The request.body should contain a standard HarperDB operation and must also include the hdb_user property that was in request.body provided in the callback.

• requestWithoutAuthentication

  Executes a request against HarperDB without any security checks around whether the inbound user is allowed to make this request. For security purposes, you should always take the following precautions when using this method:

  • Properly handle user-submitted values, including url params. User-submitted values should only be used for search_value and for defining values in records. Special care should be taken to properly escape any values if user-submitted values are used for SQL.

logger

This helper allows you to write directly to the Custom Functions log file, custom_functions.log. It’s useful for debugging during development, although you may also use the console logger. There are 5 functions contained within logger, each of which pertains to a different logging.level configuration in your harperdb-config.yaml file.

• logger.trace(‘Starting the handler for /dogs’)

• logger.debug(‘This should only fire once’)

• logger.warn(‘This should never ever fire’)

• logger.error(‘This did not go well’)

• logger.fatal(‘This did not go very well at all’)

Custom function projects can be structured and managed like normal Node.js projects. You can include external dependencies, include them in your route and helper files, and manage your revisions without changing your development tooling or pipeline.

• To initialize your project to use npm packages, use the terminal to execute npm init from the root of your project folder.

• To implement version control using git, use the terminal to execute git init from the root of your project folder.

The module can be utilized to serve static files.

Install the module in your project by running npm i @fastify/static from inside your project directory.

Register @fastify/static with the server and set root to the absolute path of the directory that contains the static files to serve.

For further information on how to send specific files see the docs.

Custom functions are a key part of building a complete HarperDB application. It is highly recommended that you use Custom Functions as the primary mechanism for your application to access your HarperDB database. Using Custom Functions gives you complete control over the accessible endpoints, how users are authenticated and authorized, what data is accessed from the database, and how it is aggregated and returned to users.

• Add your own API endpoints to a standalone API server inside HarperDB

• Use HarperDB Core methods to interact with your data at lightning speed

• Custom Functions are powered by Fastify, so they’re extremely flexible

• Manage in HarperDB Studio, or use your own IDE and Version Management System

• Distribute your Custom Functions to all your HarperDB instances with a single click

Before you get started with Custom Functions, here’s a primer on the basic configuration and the structure of a Custom Functions Project.

Configuration

Custom Functions are configured in the harperdb-config.yaml file located in the operations API root directory (by default this is a directory named hdb located in the home directory of the current user). Below is a view of the Custom Functions' section of the config YAML file, plus descriptions of important Custom Functions settings.

• enabled A boolean value that tells HarperDB to start the Custom Functions server. Set it to true to enable custom functions and false to disable. enabled is true by default.

• network.port This is the port HarperDB will use to start a standalone Fastify Server dedicated to serving your Custom Functions’ routes.

• root This is the root directory where your Custom Functions projects and their files will live. By default, it’s in your <ROOTPATH>, but you can locate it anywhere--in a developer folder next to your other development projects, for example.

Project Structure

project folder

The name of the folder that holds your project files serves as the root prefix for all the routes you create. All routes created in the dogs project folder will have a URL like this: https://my-server-url.com:9926/dogs/my/route. As such, it’s important that any project folders you create avoid any characters that aren’t URL-friendly. You should avoid URL delimiters in your folder names.

/routes folder

/helpers folder

These files are JavaScript modules that you can use in your handlers, or for custom preValidation hooks. Examples include calls to third party Authentication services, filters for results of calls to HarperDB, and custom error responses. As modules, you can use standard import and export functionality.

/static folder

If you’d like to serve your visitors a static website, you can place the html and supporting files into a directory called static. The directory must have an index.html file, and can have as many supporting resources as are necessary in whatever subfolder structure you prefer within that static directory.

To create a project using our web-based GUI, HarperDB Studio, checkout out how to manage Custom Functions .

Otherwise, to create a project, you have the following options:

1. Use the add_custom_function_project operation

   This operation creates a new project folder, and populates it with templates for the routes, helpers, and static subfolders.

1. Clone our public gitHub project template

   This requires a local installation. Remove the .git directory for a clean slate of git history.

1. Create a project folder in your Custom Functions root directory and initialize

   This requires a local installation.

Helpers are functions for use within your routes. You may want to use the same helper in multiple route files, so this allows you to write it once, and include it wherever you need it.

• To use your helpers, they must be exported from your helper file. Please use any standard export mechanisms available for your module system. We like ESM, ECMAScript Modules. Our example below exports using module.exports.

• You must import the helper module into the file that needs access to the exported functions. With ESM, you'd use a require statement. See in Define Routes.

Below is code from the customValidation helper that is referenced in . It takes the request and the logger method from the route declaration, and makes a call to an external API to validate the headers using fetch. The API in this example is just returning a list of ToDos, but it could easily be replaced with a call to a real authentication service.

One way to manage Custom Functions is through HarperDB Studio. It performs all the necessary operations automatically. To get started, navigate to your instance in HarperDB Studio and click the subnav link for “functions”. If you have not yet enabled Custom Functions, it will walk you through the process. Once configuration is complete, you can manage and deploy Custom Functions in minutes.

For any changes made to your routes, helpers, or projects, you’ll need to restart the Custom Functions server to see them take effect. HarperDB Studio does this automatically whenever you create or delete a project, or add, edit, or edit a route or helper. If you need to start the Custom Functions server yourself, you can use the following operation to do so:

{
    "operation": "restart_service",
    "service": "custom_functions"
}

HarperDB Custom Functions projects are managed by HarperDB’s process manager. As such, it may seem more difficult to debug Custom Functions than your standard project. The goal of this document is to provide best practices and recommendations for debugging your Custom Function.

For local debugging and development, it is recommended that you use standard console log statements for logging. For production use, you may want to use HarperDB's logging facilities, so you aren't logging to the console. The HarperDB Custom Functions template includes the HarperDB logger module in the primary function parameters with the name logger. This logger can be used to output messages directly to the HarperDB log using standardized logging level functions, described below. The log level can be set in the HarperDB Configuration File.

HarperDB Logger Functions

• trace(message): Write a 'trace' level log, if the configured level allows for it.

• debug(message): Write a 'debug' level log, if the configured level allows for it.

• info(message): Write a 'info' level log, if the configured level allows for it.

• warn(message): Write a 'warn' level log, if the configured level allows for it.

• error(message): Write a 'error' level log, if the configured level allows for it.

• fatal(message): Write a 'fatal' level log, if the configured level allows for it.

• notify(message): Write a 'notify' level log.

For debugging purposes, it is recommended to use notify as these messages will appear in the log regardless of log level configured.

Viewing the Log

The HarperDB Log can be found on the Studio Status page or in the local Custom Functions log file, <HDBROOT>/log/custom_functions.log. Additionally, you can use the read_log operation to query the HarperDB log.

Example 1: Execute Query and Log Results

This example performs a SQL query in HarperDB and logs the result. This example utilizes the logger.notify function to log the stringified version of the result. If an error occurs, it will output the error using logger.error and return the error.

server.route({
    url: '/',
    method: 'GET',
    handler: async (request) => {
        request.body = {
            operation: 'sql',
            sql: 'SELECT * FROM dev.dog ORDER BY dog_name'
        };

        try {
            let result = await hdbCore.requestWithoutAuthentication(request);
            logger.notify(`Query Result: ${JSON.stringify(result)}`);
            return result;
        } catch (e) {
            logger.error(`Query Error: ${e}`);
            return e;
        }
    }
});

Example 2: Execute Multiple Queries and Log Activity

This example performs two SQL queries in HarperDB with logging throughout to describe what is happening. This example utilizes the logger.notify function to log the stringified version of the operation and the result of each query. If an error occurs, it will output the error using logger.error and return the error.

server.route({
    url: '/example',
    method: 'GET',
    handler: async (request) => {
        logger.notify('/example called!');
        const results = [];

        request.body = {
            operation: 'sql',
            sql: 'SELECT * FROM dev.dog WHERE id = 1'
        };
        logger.notify(`Query 1 Operation: ${JSON.stringify(request.body)}`);
        try {
            let result = await hdbCore.requestWithoutAuthentication(request);
            logger.notify(`Query 1: ${JSON.stringify(result)}`);
            results.push(result);
        } catch (e) {
            logger.error(`Query 1: ${e}`);
            return e;
        }

        request.body = {
            operation: 'sql',
            sql: 'SELECT * FROM dev.dog WHERE id = 2'
        };
        logger.notify(`Query 2 Operation: ${JSON.stringify(request.body)}`);
        try {
            let result = await hdbCore.requestWithoutAuthentication(request);
            logger.notify(`Query 2: ${JSON.stringify(result)}`);
            results.push(result);
        } catch (e) {
            logger.error(`Query 2: ${e}`);
            return e;
        }

        logger.notify('/example complete!');
        return results;
    }
});

Library of example projects and tutorials using Custom Functions:

• Authorization in HarperDB using Okta Customer Identity Cloud, by Yitaek Hwang

• How to Speed Up your Applications by Caching at the Edge with HarperDB, by Danny Adams

• OAuth Authentication in HarperDB using Auth0 & Node.js, by Lucas Santos

• How To Create a CRUD API with Next.js & HarperDB Custom Functions, by Colby Fayock

• Build a Dynamic REST API with Custom Functions, by Terra Roush

• How to use HarperDB Custom Functions to Build your Entire Backend, by Andrew Baisden

• Using TensorFlowJS & HarperDB Custom Functions for Machine Learning, by Kevin Ashcraft

• Build & Deploy a Fitness App with Python & HarperDB, by Patrick Löber

• Create a Discord Slash Bot using HarperDB Custom Functions, by Soumya Ranjan Mohanty

• How I used HarperDB Custom Functions to Build a Web App for my Newsletter, by Hrithwik Bharadwaj

• How I used HarperDB Custom Functions and Recharts to create Dashboard, by Tapas Adhikary

• How To Use HarperDB Custom Functions With Your React App, by Ankur Tyagi

• Build a Web App Using HarperDB’s Custom Functions, livestream by Jaxon Repp

• How to Web Scrape Using Python, Snscrape & Custom Functions, by Davis David

• What’s the Big Deal w/ Custom Functions, Select* Podcast

Check out our always-expanding library of templates in our open-source HarperDB-Add-Ons GitHub repo.

One way to manage Custom Functions is through HarperDB Studio. It performs all the necessary operations automatically. To get started, navigate to your instance in HarperDB Studio and click the subnav link for “functions”. If you have not yet enabled Custom Functions, it will walk you through the process. Once configuration is complete, you can manage and deploy Custom Functions in minutes.

HarperDB Studio manages your Custom Functions using nine HarperDB operations. You may view these operations within our API Docs. A brief overview of each of the operations is below:

• custom_functions_status

  Returns the state of the Custom Functions server. This includes whether it is enabled, upon which port it is listening, and where its root project directory is located on the host machine.

• get_custom_functions

  Returns an array of projects within the Custom Functions root project directory. Each project has details including each of the files in the routes and helpers directories, and the total file count in the static folder.

• get_custom_function

  Returns the content of the specified file as text. HarperDB Studio uses this call to render the file content in its built-in code editor.

• set_custom_function

  Updates the content of the specified file. HarperDB Studio uses this call to save any changes made through its built-in code editor.

• drop_custom_function

  Deletes the specified file.

• add_custom_function_project

  Creates a new project folder in the Custom Functions root project directory. It also inserts into the new directory the contents of our Custom Functions Project template, which is available publicly, here: https://github.com/HarperDB/harperdb-custom-functions-template.

• drop_custom_function_project

  Deletes the specified project folder and all of its contents.

• package_custom_function_project

  Creates a .tar file of the specified project folder, then reads it into a base64-encoded string and returns that string the user.

• deploy_custom_function_project

  Takes the output of package_custom_function_project, decrypts the base64-encoded string, reconstitutes the .tar file of your project folder, and extracts it to the Custom Functions root project directory.