Dynamic Schema
HarperDB is built to make data ingestion simple. A primary driver of that is the Dynamic Schema. The purpose of this document is to provide a detailed explanation of the dynamic schema specifically related to schema definition and data ingestion.
The dynamic schema provides the structure of schema and table namespaces while simultaneously providing the flexibility of a data-defined schema. Individual attributes are reflexively created as data is ingested, meaning the table will adapt to the structure of data ingested. HarperDB tracks the metadata around schemas, tables, and attributes allowing for describe table, describe schema, and describe all operations.
Schemas
HarperDB schemas are analogous to a namespace that groups tables together. A schema is required to create a table.
Tables
HarperDB tables group records together with a common data pattern. To create a table users must provide a table name and a primary key.
Table Name: Used to identify the table.
Primary Key: This is a required attribute that serves as the unique identifier for a record and is also known as the
hash_attribute
in HarperDB.
Primary Key
The primary key (also referred to as the hash_attribute
) is used to uniquely identify records. Uniqueness is enforced on the primary; inserts with the same primary key will be rejected. If a primary key is not provided on insert, a GUID will be automatically generated and returned to the user. The HarperDB Storage Algorithm utilizes this value for indexing.
Standard Attributes
Additional attributes are reflexively added via insert and update operations (in both SQL and NoSQL) when new attributes are included in the data structure provided to HarperDB. As a result, schemas are additive, meaning new attributes are created in the underlying storage algorithm as additional data structures are provided. HarperDB offers create_attribute
and drop_attribute
operations for users who prefer to manually define their data model independent of data ingestion. When new attributes are added to tables with existing data the value of that new attribute will be assumed null
for all existing records.
Audit Attributes
HarperDB automatically creates two audit attributes used on each record.
__createdtime__
: The time the record was created in Unix Epoch with milliseconds format.__updatedtime__
: The time the record was updated in Unix Epoch with milliseconds format.
Dynamic Schema Example
To better understand the behavior let’s take a look at an example. This example utilizes HarperDB API operations.
Create a Schema
Create a Table
Notice the schema name, table name, and hash attribute name are the only required parameters.
At this point the table does not have structure beyond what we provided, so the table looks like this:
dev.dog
![](https://docs.harperdb.io/~gitbook/image?url=https%3A%2F%2F2630814040-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FswpiK7SGxIxJgTWi32sK%252Fuploads%252Fgit-blob-b3cdee75b1d1248816143a003c82c6b421e7afd1%252Fdynamic_schema_2_create_table.png.webp%3Falt%3Dmedia&width=768&dpr=4&quality=100&sign=8e5a170e&sv=1)
Insert Record
To define attributes we do not need to do anything beyond sending them in with an insert operation.
With a single record inserted and new attributes defined, our table now looks like this:
dev.dog
![](https://docs.harperdb.io/~gitbook/image?url=https%3A%2F%2F2630814040-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FswpiK7SGxIxJgTWi32sK%252Fuploads%252Fgit-blob-4f79c0f4e2b2d3f0cc1763d364b9bbed532b2808%252Fdynamic_schema_3_insert_record.png.webp%3Falt%3Dmedia&width=768&dpr=4&quality=100&sign=2f8d3ae4&sv=1)
Indexes have been automatically created for dog_name
and owner_name
attributes.
Insert Additional Record
If we continue inserting records with the same data schema no schema updates are required. One record will omit the hash attribute from the insert to demonstrate GUID generation.
In this case, there is no change to the schema. Our table now looks like this:
dev.dog
![](https://docs.harperdb.io/~gitbook/image?url=https%3A%2F%2F2630814040-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FswpiK7SGxIxJgTWi32sK%252Fuploads%252Fgit-blob-2caa7e75a72cdbd1ead094c0d49c4178bd7856d1%252Fdynamic_schema_4_insert_additional_record.png.webp%3Falt%3Dmedia&width=768&dpr=4&quality=100&sign=fd8e6262&sv=1)
Update Existing Record
In this case, we will update a record with a new attribute not previously defined on the table.
Now we have a new attribute called weight_lbs
. Our table now looks like this:
dev.dog
![](https://docs.harperdb.io/~gitbook/image?url=https%3A%2F%2F2630814040-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FswpiK7SGxIxJgTWi32sK%252Fuploads%252Fgit-blob-041a71d7d7f54f4c4a7fac39a5ec88b1b86b8de1%252Fdynamic_schema_5_update_existing_record.png.webp%3Falt%3Dmedia&width=768&dpr=4&quality=100&sign=f3494544&sv=1)
Query Table with SQL
Now if we query for all records where weight_lbs
is null
we expect to get back two records.
This results in the expected two records being returned.
![](https://docs.harperdb.io/~gitbook/image?url=https%3A%2F%2F2630814040-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FswpiK7SGxIxJgTWi32sK%252Fuploads%252Fgit-blob-95dac39c2f257f4f881dce013525875164979995%252Fdynamic_schema_6_query_table_with_sql.png.webp%3Falt%3Dmedia&width=768&dpr=4&quality=100&sign=ced1c2b3&sv=1)
Last updated