LogoLogo
4.1
4.1
  • Developer Documentation
  • Install HarperDB
    • On Linux
  • Getting Started
  • Full API Documentation
  • HarperDB Studio
    • Create an Account
    • Log In & Password Reset
    • Resources (Marketplace, Drivers, Tutorials, & Example Code)
    • Organizations
    • Instances
    • Query Instance Data
    • Manage Schemas / Browse Data
    • Manage Charts
    • Manage Clustering
    • Manage Instance Users
    • Manage Instance Roles
    • Manage Functions
    • Instance Metrics
    • Instance Configuration
    • Instance Example Code
    • Enable Mixed Content
  • HarperDB Cloud
    • IOPS Impact on Performance
    • Instance Size Hardware Specs
    • Alarms
    • Verizon 5G Wavelength
  • Security
    • JWT Authentication
    • Basic Authentication
    • Configuration
    • Users & Roles
  • Clustering
    • Requirements and Definitions
    • Creating A Cluster User
    • Naming A Node
    • Enabling Clustering
    • Establishing Routes
    • Subscription Overview
    • Managing Subscriptions
    • Things Worth Knowing
  • Custom Functions
    • Requirements and Definitions
    • Create a Project
    • Define Routes
    • Define Helpers
    • Host A Static Web UI
    • Using NPM and GIT
    • Custom Functions Operations
    • Restarting the Server
    • Debugging a Custom Function
    • Custom Functions Templates
    • Example Projects
  • Add-ons and SDKs
    • Google Data Studio
  • SQL Guide
    • SQL Features Matrix
    • Insert
    • Update
    • Delete
    • Select
    • Joins
    • SQL Date Functions
    • SQL Reserved Word
    • SQL Functions
    • SQL JSON Search
    • SQL Geospatial Functions
      • geoArea
      • geoLength
      • geoDifference
      • geoDistance
      • geoNear
      • geoContains
      • geoEqual
      • geoCrosses
      • geoConvert
  • HarperDB CLI
  • Configuration File
  • Logging
  • Transaction Logging
  • Audit Logging
  • Jobs
  • Upgrade a HarperDB Instance
  • Reference
    • Storage Algorithm
    • Dynamic Schema
    • Data Types
    • Content Types/Data Formats
    • HarperDB Headers
    • HarperDB Limits
  • Support
  • Release Notes
    • HarperDB Tucker (Version 4)
      • 4.1.0
      • 4.0.6
      • 4.0.5
      • 4.0.4
      • 4.0.3
      • 4.0.2
      • 4.0.1
      • 4.0.0
    • HarperDB Monkey (Version 3)
      • 3.3.0
      • 3.2.1
      • 3.2.0
      • 3.1.5
      • 3.1.4
      • 3.1.3
      • 3.1.2
      • 3.1.1
      • 3.1.0
      • 3.0.0
    • HarperDB Penny (Version 2)
      • 2.3.1
      • 2.3.0
      • 2.2.3
      • 2.2.2
      • 2.2.0
      • 2.1.1
    • HarperDB Alby (Version 1)
      • 1.3.1
      • 1.3.0
      • 1.2.0
      • 1.1.0
Powered by GitBook

© HarperDB. All Rights Reserved

On this page
  • Clustering Logging
  • Log File vs Standard Streams
  • Logging Rotation
  • Read Logs via the API
Export as PDF

Logging

HarperDB maintains a log of events that take place throughout operation. Log messages can be used for diagnostics purposes as well as monitoring.

All logs (except for the install log) are stored in the main log file in the hdb directory <ROOTPATH>/log/hdb.log. The install log is located in the HarperDB application directory most likely located in your npm directory npm/harperdb/logs.

Each log message has several key components for consistent reporting of events. A log message has a format of:

<timestamp> [<level>] [<thread/id>] ...[<other tags>]: <message>

For example, a typical log entry looks like:

2023-03-09T14:25:05.269Z [notify] [main/0]: HarperDB successfully started.

The components of a log entry are:

  • timestamp - This is the date/time stamp when the event occurred

  • level - This is an associated log level that gives a rough guide to the importance and urgency of the message. The available log levels in order of least urgent (and more verbose) are: trace, debug, info, warn, error, fatal, and notify.

  • thread/id - This reports the name of the thread and the thread id, that the event was reported on. Note that NATS logs are recorded by their process name and there is no thread id for them since they are a separate process. Key threads are:

    • main - This is the thread that is responsible for managing all other threads and routes incoming requests to the other threads

    • http - These are the worker threads that handle the primary workload of incoming HTTP requests to the operations API and custom functions.

    • Clustering* - These are threads and processes that handle replication.

    • job - These are job threads that have been started to handle operations that are executed in a separate job thread.

  • tags - Logging from a custom function will include a "custom-function" tag in the log entry. Most logs will not have any additional tags.

  • message - This is the main message that was reported.

We try to keep logging to a minimum by default, to do this the default log level is error. If you require more information from the logs, increasing the log level down will provide that.

The log level can be changed by modifying logging.level in the config file harperdb-config.yaml.

Clustering Logging

Log File vs Standard Streams

HarperDB logs can optionally be streamed to standard streams. Logging to standard streams (stdout/stderr) is primarily used for container logging drivers. For more traditional installations, we recommend logging to a file. Logging to both standard streams and to a file can be enabled simultaneously. To log to standard streams effectively, make sure to directly run harperdb and don't start it as a separate process (don't use harperdb start) and logging.stdStreams must be set to true. Note, logging to standard streams only will disable clustering catchup.

Logging Rotation

Read Logs via the API

To access specific logs you may query the HarperDB API. Logs can be queried using the read_log operation. read_log returns outputs from the log based on the provided search criteria.

{
    "operation": "read_log",
    "start": 0,
    "limit": 1000,
    "level": "error",
    "from": "2021-01-25T22:05:27.464+0000",
    "until": "2021-01-25T23:05:27.464+0000",
    "order": "desc"
}
PreviousConfiguration FileNextTransaction Logging

Last updated 1 year ago

HarperDB clustering utilizes two servers, named Hub and Leaf. The Hub server is responsible for establishing the mesh network that connects instances of HarperDB and the Leaf server is responsible for managing the message stores (streams) that replicate and store messages between instances. Due to the verbosity of these servers there is a separate log level configuration for them. To adjust their log verbosity set clustering.logLevel in the config file harperdb-config.yaml. Valid log levels from least verbose are error, warn, info, debug and trace.

Log rotation allows for managing log files, such as compressing rotated log files, archiving old log files, determining when to rotate, and the like. This will allow for organized storage and efficient use of disk space. For more information see “logging” in our .

Nats
config docs