LogoLogo
Studio
4.5
4.5
  • Harper Docs
  • Getting Started
  • Developers
    • Applications
      • Caching
      • Defining Schemas
      • Defining Roles
      • Debugging Applications
      • Define Fastify Routes
      • Web Applications
      • Example Projects
    • Components
      • Managing
      • Reference
      • Built-In Components
    • REST
    • Operations API
      • Quick Start Examples
      • Databases and Tables
      • NoSQL Operations
      • Bulk Operations
      • Users and Roles
      • Clustering
        • Clustering with NATS
      • Custom Functions
      • Components
      • Registration
      • Jobs
      • Logs
      • Utilities
      • Token Authentication
      • SQL Operations
      • Advanced JSON SQL Examples
    • Real-Time
    • Replication/Clustering
      • Sharding
      • Legacy NATS Clustering
        • Requirements and Definitions
        • Creating A Cluster User
        • Naming A Node
        • Enabling Clustering
        • Establishing Routes
        • Subscription Overview
        • Managing Subscriptions
        • Things Worth Knowing
        • Certificate Management
    • Security
      • JWT Authentication
      • Basic Authentication
      • mTLS Authentication
      • Configuration
      • Users & Roles
      • Certificate Management
    • SQL Guide
      • SQL Features Matrix
      • SQL Date Functions
      • SQL Reserved Word
      • SQL Functions
      • SQL JSON Search
      • SQL Geospatial Functions
    • Miscellaneous
      • Google Data Studio
      • SDKs
      • Query Optimization
  • Administration
    • Best Practices and Recommendations
    • Logging
      • Standard Logging
      • Audit Logging
      • Transaction Logging
    • Clone Node
    • Compact
    • Jobs
    • Harper Studio
      • Create an Account
      • Log In & Password Reset
      • Organizations
      • Instances
      • Query Instance Data
      • Manage Databases / Browse Data
      • Manage Clustering
      • Manage Instance Users
      • Manage Instance Roles
      • Manage Applications
      • Instance Metrics
      • Instance Configuration
      • Enable Mixed Content
  • Deployments
    • Configuration File
    • Harper CLI
    • Install Harper
      • On Linux
    • Upgrade a Harper Instance
    • Harper Cloud
      • IOPS Impact on Performance
      • Instance Size Hardware Specs
      • Alarms
      • Verizon 5G Wavelength
  • Technical Details
    • Reference
      • Analytics
      • Architecture
      • Content Types
      • Data Types
      • Dynamic Schema
      • GraphQL
      • Harper Headers
      • Harper Limits
      • Globals
      • Resource Class
      • Transactions
      • Storage Algorithm
      • Blob
    • Release Notes
      • Harper Tucker (Version 4)
        • 4.5.8
        • 4.5.7
        • 4.5.6
        • 4.5.5
        • 4.5.4
        • 4.5.3
        • 4.5.2
        • 4.5.1
        • 4.5.0
        • 4.4.24
        • 4.4.23
        • 4.4.22
        • 4.4.21
        • 4.4.20
        • 4.4.19
        • 4.4.18
        • 4.4.17
        • 4.4.16
        • 4.4.15
        • 4.4.14
        • 4.4.13
        • 4.4.12
        • 4.4.11
        • 4.4.10
        • 4.4.9
        • 4.4.8
        • 4.4.7
        • 4.4.6
        • 4.4.5
        • 4.4.4
        • 4.4.3
        • 4.4.2
        • 4.4.1
        • 4.4.0
        • 4.3.38
        • 4.3.37
        • 4.3.36
        • 4.3.35
        • 4.3.34
        • 4.3.33
        • 4.3.32
        • 4.3.31
        • 4.3.30
        • 4.3.29
        • 4.3.28
        • 4.3.27
        • 4.3.26
        • 4.3.25
        • 4.3.24
        • 4.3.23
        • 4.3.22
        • 4.3.21
        • 4.3.20
        • 4.3.19
        • 4.3.18
        • 4.3.17
        • 4.3.16
        • 4.3.15
        • 4.3.14
        • 4.3.13
        • 4.3.12
        • 4.3.11
        • 4.3.10
        • 4.3.9
        • 4.3.8
        • 4.3.7
        • 4.3.6
        • 4.3.5
        • 4.3.4
        • 4.3.3
        • 4.3.2
        • 4.3.1
        • 4.3.0
        • 4.2.8
        • 4.2.7
        • 4.2.6
        • 4.2.5
        • 4.2.4
        • 4.2.3
        • 4.2.2
        • 4.2.1
        • 4.2.0
        • 4.1.2
        • 4.1.1
        • 4.1.0
        • 4.0.7
        • 4.0.6
        • 4.0.5
        • 4.0.4
        • 4.0.3
        • 4.0.2
        • 4.0.1
        • 4.0.0
        • Tucker
      • HarperDB Monkey (Version 3)
        • 3.3.0
        • 3.2.1
        • 3.2.0
        • 3.1.5
        • 3.1.4
        • 3.1.3
        • 3.1.2
        • 3.1.1
        • 3.1.0
        • 3.0.0
      • HarperDB Penny (Version 2)
        • 2.3.1
        • 2.3.0
        • 2.2.3
        • 2.2.2
        • 2.2.0
        • 2.1.1
      • HarperDB Alby (Version 1)
        • 1.3.1
        • 1.3.0
        • 1.2.0
        • 1.1.0
  • More Help
    • Support
    • Slack
    • Contact Us
Powered by GitBook
On this page
  1. Technical Details
  2. Reference

Analytics

Harper provides extensive telemetry and analytics data to help monitor the status of the server and work loads, and to help understand traffic and usage patterns to identify issues and scaling needs, and identify queries and actions that are consuming the most resources.

Harper collects statistics for all operations, URL endpoints, and messaging topics, aggregating information by thread, operation, resource, and methods, in real-time. These statistics are logged in the hdb_raw_analytics and hdb_analytics table in the system database.

There are two "levels" of analytics in the Harper analytics table: the first is the immediate level of raw direct logging of real-time statistics. These analytics entries are recorded once a second (when there is activity) by each thread, and include all recorded activity in the last second, along with system resource information. The records have a primary key that is the timestamp in milliseconds since epoch. This can be queried (with superuser permission) using the search_by_conditions operation (this will search for 10 seconds worth of analytics) on the hdb_raw_analytics table:

POST http://localhost:9925
Content-Type: application/json

{
    "operation": "search_by_conditions",
    "schema": "system",
    "table": "hdb_raw_analytics",
    "conditions": [{
        "search_attribute": "id",
        "search_type": "between",
        "search_value": [168859400000, 1688594010000]
    }]
}

And a typical response looks like:

{
    "time": 1688594390708,
    "period": 1000.8336279988289,
    "metrics": [
      {
        "metric": "bytes-sent",
        "path": "search_by_conditions",
        "type": "operation",
        "median": 202,
        "mean": 202,
        "p95": 202,
        "p90": 202,
        "count": 1
      },
      ...
      {
        "metric": "memory",
        "threadId": 2,
        "rss": 1492664320,
        "heapTotal": 124596224,
        "heapUsed": 119563120,
        "external": 3469790,
        "arrayBuffers": 798721
      },
      {
        "metric": "utilization",
        "idle": 138227.52767700003,
        "active": 70.5066209952347,
        "utilization": 0.0005098165086230495
      }
    ],
    "threadId": 2,
    "totalBytesProcessed": 12182820,
    "id": 1688594390708.6853
}

The second level of analytics recording is aggregate data. The aggregate records are recorded once a minute, and aggregate the results from all the per-second entries from all the threads, creating a summary of statistics once a minute. The ids for these milliseconds since epoch can be queried from the hdb_analytics table. You can query these with an operation like:

POST http://localhost:9925
Content-Type: application/json

{
    "operation": "search_by_conditions",
    "schema": "system",
    "table": "hdb_analytics",
    "conditions": [{
        "search_attribute": "id",
        "search_type": "between",
        "search_value": [1688194100000, 1688594990000]
    }]
}

And a summary record looks like:

{
    "period": 60000,
    "metric": "bytes-sent",
    "method": "connack",
    "type": "mqtt",
    "median": 4,
    "mean": 4,
    "p95": 4,
    "p90": 4,
    "count": 1,
    "id": 1688589569646,
    "time": 1688589569646
}

The following are general resource usage statistics that are tracked:

  • memory - This includes RSS, heap, buffer and external data usage.

  • utilization - How much of the time the worker was processing requests.

  • mqtt-connections - The number of MQTT connections.

The following types of information is tracked for each HTTP request:

  • success - How many requests returned a successful response (20x response code). TTFB - Time to first byte in the response to the client.

  • transfer - Time to finish the transfer of the data to the client.

  • bytes-sent - How many bytes of data were sent to the client.

Requests are categorized by operation name, for the operations API, by the resource (name) with the REST API, and by command for the MQTT interface.

PreviousReferenceNextArchitecture

Last updated 1 month ago