Configuration File
HarperDB is configured through a YAML file called
harperdb-config.yaml
located in the operations API root directory (by default this is a directory named hdb
located in the home directory of the current user).All available configuration will be populated by default in the config file on install, regardless of whether it is used.
The configuration elements in
harperdb-config.yaml
use camelcase: operationsApi
.To change a configuration value edit the
harperdb-config.yaml
file and save any changes. HarperDB must be restarted for changes to take effect.Alternately, configuration can be changed via environment and/or command line variables or via the API. To access lower level elements, use underscores to append parent/child elements (when used this way elements are case insensitive):
- Environment variables: `OPERATIONSAPI_NETWORK_PORT=9925`
- Command line variables: `--OPERATIONSAPI_NETWORK_PORT 9925`
- Calling `set_configuration` through the API: `operationsApi_network_port: 9925`
The
clustering
section configures the clustering engine, this is used to replicate data between instances of HarperDB.Clustering offers a lot of different configurations, however in a majority of cases the only options you will need to pay attention to are:
clustering.enabled
Enable the clustering processes.clustering.hubServer.cluster.network.port
The port other nodes will connect to. This port must be accessible from other cluster nodes.clustering.hubServer.cluster.network.routes
The connections to other instances.clustering.nodeName
The name of your node, must be unique within the cluster.clustering.user
The name of the user credentials used for Inter-node authentication.
enabled
- Type: boolean; Default: falseEnable clustering.
Note: If you enabled clustering but do not create and add a cluster user you will get a validation error. See
user
description below on how to add a cluster user.clustering:
enabled: true
clustering.hubServer.cluster
Clustering’s
hubServer
facilitates the HarperDB mesh network and discovery service.clustering:
hubServer:
cluster:
name: harperdb
network:
port: 9932
routes:
- host: 3.62.184.22
port: 9932
- host: 3.735.184.8
port: 9932
name
- Type: string, Default: harperdbThe name of your cluster. This name needs to be consistent for all other nodes intended to be meshed in the same network.
port
- Type: integer, Default: 9932The port the hub server uses to accept cluster connections
routes
- Type: array, Default: nullAn object array that represent the host and port this server will cluster to. Each object must have two properties
port
and host
. Multiple entries can be added to create network resiliency in the event one server is unavailable. Routes can be added, updated and removed either by directly editing the harperdb-config.yaml
file or by using the cluster_set_routes
or cluster_delete_routes
API endpoints.host
- Type: stringThe host of the remote instance you are creating the connection with.
port
- Type: integerThe port of the remote instance you are creating the connection with. This is likely going to be the
clustering.hubServer.cluster.network.port
on the remote instance.clustering.hubServer.leafNodes
clustering:
hubServer:
leafNodes:
network:
port: 9931
port
- Type: integer; Default: 9931The port the hub server uses to accept leaf server connections.
clustering.hubServer.network
clustering:
hubServer:
network:
port: 9930
port
- Type: integer; Default: 9930Use this port to connect a client to the hub server, for example using the NATs SDK to interact with the server.
clustering.leafServer.network
Manages streams, streams are ‘message stores’ that store table transactions.
clustering:
leafServer:
network:
port: 9940
routes:
- host: 3.62.184.22
port: 9931
- host: 3.735.184.8
port: 9931
port
- Type: integer; Default: 9940Use this port to connect a client to the leaf server, for example using the NATs SDK to interact with the server.
routes
- Type: array; Default: nullAn object array that represent the host and port the leaf node will directly connect with. Each object must have two properties
port
and host
. Unlike the hub server, the leaf server will establish connections to all listed hosts. Routes can be added, updated and removed either by directly editing the harperdb-config.yaml
file or by using the cluster_set_routes
or cluster_delete_routes
API endpoints.host
- Type: stringThe host of the remote instance you are creating the connection with.
port
- Type: integerThe port of the remote instance you are creating the connection with. This is likely going to be the
clustering.hubServer.leafNodes.network.port
on the remote instance.
nodeName
- Type: string; Default: nullThe name of this node in your HarperDB cluster topology. This must be a value unique from the rest of the cluster node names.
Note: If you want to change the node name make sure there are no subscriptions in place before doing so. After the name has been changed a full restart is required.
clustering:
nodeName: great_node
tls
Transport Layer Security default values are automatically generated on install.
clustering:
tls:
certificate: ~/hdb/keys/certificate.pem
certificateAuthority: ~/hdb/keys/ca.pem
privateKey: ~/hdb/keys/privateKey.pem
insecure: true
certificate
- Type: string; Default: <ROOTPATH>/keys/certificate.pemPath to the certificate file.
certificateAuthority
- Type: string; Default: <ROOTPATH>/keys/ca.pemPath to the certificate authority file.
privateKey
- Type: string; Default: <ROOTPATH>/keys/privateKey.pemPath to the private key file.
insecure
- Type: boolean; Default: trueWhen true, will skip certificate verification. For use only with self-signed certs.
user
- Type: string; Default: nullThe username given to the
cluster_user
. All instances in a cluster must use the same clustering user credentials (matching username and password).Inter-node authentication takes place via a special HarperDB user role type called
cluster_user
.The user can be created either through the API using an
add_user
request with the role set to cluster_user
, or on install using environment variables CLUSTERING_USER=cluster_person
CLUSTERING_PASSWORD=pass123!
or CLI variables harperdb --CLUSTERING_USER cluster_person
--CLUSTERING_PASSWORD
pass123!
clustering:
user: cluster_person
The
customFunctions
section configures HarperDB Custom Functions.enabled
- Type: boolean; Default: trueEnable the Custom Function server or not.
customFunctions:
enabled: true
customFunctions.network
customFunctions:
network:
cors: true
corsAccessList:
- null
headersTimeout: 60000
https: false
keepAliveTimeout: 5000
port: 9926
timeout: 120000
cors
- Type: boolean; Default: trueEnable Cross Origin Resource Sharing, which allows requests across a domain.
corsAccessList
- Type: array; Default: nullAn array of allowable domains with CORS
headersTimeout
- Type: integer; Default: 60,000 milliseconds (1 minute)Limit the amount of time the parser will wait to receive the complete HTTP headers with.
https
- Type: boolean; Default: falseEnables HTTPS on the Custom Functions API. This requires a valid certificate and key. If
false
, Custom Functions will run using standard HTTP.keepAliveTimeout
- Type: integer; Default: 5,000 milliseconds (5 seconds)Sets the number of milliseconds of inactivity the server needs to wait for additional incoming data after it has finished processing the last response.
port
- Type: integer; Default: 9926The port used to access the Custom Functions server.
timeout
- Type: integer; Default: Defaults to 120,000 milliseconds (2 minutes)The length of time in milliseconds after which a request will timeout.
nodeEnv
- Type: string; Default: productionAllows you to specify the node environment in which application will run.
customFunctions:
nodeEnv: production
production
native node logging is kept to a minimum; more caching to optimize performance. This is the default value.development
more native node logging; less caching.
root
- Type: string; Default: <ROOTPATH>/custom_functionsThe path to the folder containing Custom Function files.
customFunctions:
root: ~/hdb/custom_functions
tls
Transport Layer SecuritycustomFunctions:
tls:
certificate: ~/hdb/keys/certificate.pem
certificateAuthority: ~/hdb/keys/ca.pem
privateKey: ~/hdb/keys/privateKey.pem
certificate
- Type: string; Default: <ROOTPATH>/keys/certificate.pemPath to the certificate file.
certificateAuthority
- Type: string; Default: <ROOTPATH>/keys/ca.pemPath to the certificate authority file.
privateKey
- Type: string; Default: <ROOTPATH>/keys/privateKey.pemPath to the private key file.
The
ipc
section configures the HarperDB Inter-Process Communication interface.ipc:
network:
port: 9383
port
- Type: integer; Default: 9383The port the IPC server runs on. The default is
9383
.
The
localStudio
section configures the locally HarperDB Studio. A simplified GUI for HarperDB hosted on the server. A more comprehensive GUI is hosted by HarperDB at https://studio.harperdb.io. Note, all database traffic from either localStudio
or HarperDB Studio is made directly from your browser to the instance.enabled
- Type: boolean; Default: falseEnabled the local studio or not.
localStudio:
enabled: false
The
logging
section configures HarperDB logging across all HarperDB functionality. HarperDB leverages pm2 for logging. Each process group gets their own log file which is located in logging.root
.auditLog
- Type: boolean; Default: falseEnabled table transaction logging.
logging:
auditLog: false
To access the audit logs, use the API operation
read_audit_log
. It will provide a history of the data, including original records and changes made, in a specified table.{
"operation": "read_audit_log",
"schema": "dev",
"table": "dog"
}
file
- Type: boolean; Default: trueDefines whether or not to log to a file.
logging:
file: true
level
- Type: string; Default: errorControl the verbosity of logs.
logging:
level: error
There exists a log level hierarchy in order as
trace
, debug
, info
, warn
, error
, fatal
, and notify
. When the level is set to trace
logs will be created for all possible levels. Whereas if the level is set to fatal
, the only entries logged will be fatal
and notify
. The default value is error
.root
- Type: string; Default: <ROOTPATH>/logThe path where the log files will be written.
logging:
root: ~/hdb/log
rotation
Logging rotation is managed by a pm2 module called logrotate. In order to enable rotation you must set
rotate
to true.logging:
rotation:
compress: false
dateFormat: YYYY-MM-DD_HH-mm-ss
maxSize: 10M
retain: 30
rotate: false
rotateInterval: 0 0 * * *
rotateModule: true
timezone: GMT
workerInterval: 30
compress
- Type: boolean; Default: falseEnables compression via gzip when logs are rotated.
dateFormat
- Type: string; Default: YYY-MM-DD_HH-mm-ssFormat of the date used to name the log file.
maxSize
- Type: string; Default: 10MSets the value for when the logging file will rotate when it exceeds this set value. Must use units M(egabyte), G(igabyte), or K(ilobyte).
retain
- Type: integer; Default: 30Sets the number of rotated logs that are kept at any one time. For example, if set at 30, there will be 30 rotated logs plus your current one.
rotate
- Type: boolean; Default: falseEnables logging rotation.
rotateInterval
- Type: string; Default: 0 0 * * *rotateModule
- Type: boolean; Default: trueWill rotate the pm2 module.
timezone
- Type: string; Default: GMTSets the timezone for logging rotation.
workerInterval
- Type: integer; Default: 30Sets the interval at which the worker checks log size in seconds.
stdStreams
- Type: boolean; Default: falseLog HarperDB logs to the standard output and error streams. The
operationsApi.foreground
flag must be enabled in order to receive the stream.logging:
stdStreams: false
The
operationsApi
section configures the HarperDB Operations API.authentication
operationsApi:
authentication:
operationTokenTimeout: 1d
refreshTokenTimeout: 30d
operationTokenTimeout
- Type: string; Default: 1dDefines the length of time an operation token will be valid until it expires. Example values: https://github.com/vercel/ms.
refreshTokenTimeout
- Type: string; Default: 1dDefines the length of time a refresh token will be valid until it expires. Example values: https://github.com/vercel/ms.
foreground
- Type: boolean; Default: falseDetermines whether or not HarperDB runs in the foreground.
operationsApi:
foreground: false
network
operationsApi:
network:
cors: true
corsAccessList:
- null
headersTimeout: 60000
https: false
keepAliveTimeout: 5000
port: 9925
timeout: 120000
cors
- Type: boolean; Default: trueEnable Cross Origin Resource Sharing, which allows requests across a domain.
corsAccessList
- Type: array; Default: nullAn array of allowable domains with CORS
headersTimeout
- Type: integer; Default: 60,000 milliseconds (1 minute)Limit the amount of time the parser will wait to receive the complete HTTP headers with.
https
- Type: boolean; Default: falseEnable HTTPS on the HarperDB operations endpoint. This requires a valid certificate and key. If
false
, HarperDB will run using standard HTTP.keepAliveTimeout
- Type: integer; Default: 5,000 milliseconds (5 seconds)Sets the number of milliseconds of inactivity the server needs to wait for additional incoming data after it has finished processing the last response.
port
- Type: integer; Default: 9925The port the HarperDB operations API interface will listen on.
timeout
- Type: integer; Default: Defaults to 120,000 milliseconds (2 minutes)The length of time in milliseconds after which a request will timeout.
nodeEnv
- Type: string; Default: productionAllows you to specify the node environment in which application will run.
operationsApi:
nodeEnv: production
production
native node logging is kept to a minimum; more caching to optimize performance. This is the default value.development
more native node logging; less caching.
tls
This configures the Transport Layer Security for HTTPS support.
operationsApi:
tls:
certificate: ~/hdb/keys/certificate.pem
certificateAuthority: ~/hdb/keys/ca.pem
privateKey: ~/hdb/keys/privateKey.pem
certificate
- Type: string; Default: <ROOTPATH>/keys/certificate.pemPath to the certificate file.
certificateAuthority
- Type: string; Default: <ROOTPATH>/keys/ca.pemPath to the certificate authority file.
privateKey
- Type: string; Default: <ROOTPATH>/keys/privateKey.pemPath to the private key file.
threads
- Type: number; Default: One less than the number of logical cores/ processorsThe
threads
option specifies the number of threads that will be used to service the HTTP requests for the operations API and custom functions. Generally, this should be close to the number of CPU logical cores/processors to ensure the CPU is fully utilized (a little less because HarperDB does have other threads at work), assuming HarperDB is the main service on a server.http:
threads: 11
rootPath
- Type: string; Default: home directory of the current userThe HarperDB database and applications/API/interface are decoupled from each other. The
rootPath
directory specifies where the HarperDB application persists data, config, logs, and Custom Functions.rootPath: /Users/jonsnow/hdb
writeAsync
- Type: boolean; Default: falseThe
writeAsync
option turns off disk flushing/syncing, allowing for faster write operation throughput. However, this does not provide storage integrity guarantees, and if a server crashes, it is possible that there may be data loss requiring restore from another backup/another node.storage:
writeAsync: false
caching
- Type: boolean; Default: trueThe
caching
option enables in-memory caching of records, providing faster access to frequently accessed objects. This can incur some extra overhead for situations where reads are extremely random and don't benefit from caching.storage:
caching: true
compression
- Type: boolean; Default: falseThe
compression
option enables compression of records in the database. This can be helpful for very large databases in reducing storage requirements and potentially allowing more data to be cached. This uses the very fast LZ4 compression algorithm, but this still incurs extra costs for compressing and decompressing.storage:
compression: false
noReadAhead
- Type: boolean; Default: trueThe
noReadAhead
option advises the operating system to not read ahead when reading from the database. This provides better memory utilization, except in situations where large records are used or frequent range queries are used.storage:
noReadAhead: true
prefetchWrites
- Type: boolean; Default: trueThe
prefetchWrites
option loads data prior to write transactions. This should be enabled for databases that are larger than memory (although it can be faster to disable this for smaller databases).storage:
prefetchWrites: true
Last modified 2mo ago