Add Full-Text Search to a Product Catalog

Added in: v5.3.0

This guide adds relevance-ranked text search to a product catalog, then exposes it through Harper's JavaScript and REST APIs.

What you will build

A BM25-ranked index over product names, descriptions, and tags
A Harper application endpoint with highlights and a structured category filter
REST queries for terms, phrases, and record autocomplete

Prerequisites

Harper 5.3 or later using RocksDB
A component with a GraphQL schema, JavaScript resources, and rest: true in its config.yaml

1. Declare the index

Declare an @fullText index over the fields customers search on a RocksDB-backed table. Ordinary tables inherit enabled audit logging by default, but a new full-text table requires explicit audit: true for its transaction-log recovery source. See the audit requirement for existing tables.

schema.graphql
type Product @table(database: "catalog", audit: true) @export {
	id: ID @primaryKey
	name: String
	description: String
	tags: [String]
	category: String @indexed
	price: Float @indexed
	available: Boolean
	catalogSearch: FullText
		@fullText(
			fields: [
				{ name: "name", weight: 3, highlight: true }
				{ name: "description", weight: 1, highlight: true }
				{ name: "tags" }
			]
			highlighting: { maxFragments: 2, fragmentLength: 120 }
		)
}

The query-only field name catalogSearch identifies the index. It is not stored with each product. weight: 3 makes a match in name contribute more to BM25 relevance than the same match in description or tags. Highlighting is enabled only for the fields shown to customers.

The @indexed declarations create ordinary Harper secondary indexes. Full-text searches can use them to narrow candidates, and Harper checks structured conditions against current product records before returning results.

2. Add records

Use the generated REST endpoint:

POST /Product/
Content-Type: application/json

{
	"id": "shoe-1",
	"name": "Waterproof trail running shoe",
	"description": "Lightweight shoe for wet mountain trails",
	"tags": ["outdoor", "trail"],
	"category": "footwear",
	"price": 129,
	"available": true
}

Harper commits the record first, then applies the committed change to the local derived full-text index. Each replica performs the same derivation after it has received both the full-text schema declaration and the record transactions.

If the index remains behind for more than 30 seconds, a local write can return retryable 503 code DERIVED_INDEX_LAGGING. Retry writes with backoff while the index catches up; freshness options apply only to searches. See write backpressure.

3. Search from a Harper application

resources.js
export class SearchProducts extends Resource {
	static async post(_target, data, context) {
		if (!context?.user) {
			const error = new Error('Authentication required');
			error.statusCode = 401;
			throw error;
		}
		const body = (await data) ?? {};
		const text = typeof body.q === 'string' ? body.q.trim() : '';
		if (!text) {
			const error = new Error('q is required');
			error.statusCode = 400;
			throw error;
		}
		const conditions = [
			{
				attribute: 'catalogSearch',
				comparator: 'matches',
				value: text,
				fields: ['name', 'description', 'tags'],
			},
			{ attribute: 'available', comparator: 'equals', value: true },
		];
		const category = body.category;
		if (category) conditions.push({ attribute: 'category', comparator: 'equals', value: category });
		return databases.catalog.Product.search(
			{
				conditions,
				select: ['id', 'name', 'price', '$score', '$highlights'],
				limit: 20,
				checkPermission: true,
			},
			context
		);
	}
}

Call the resource with an authenticated request:

POST /SearchProducts/
Content-Type: application/json

{ "q": "trail", "category": "footwear" }

This custom resource accepts a structured JSON search body through POST. The category filter is optional; q is required. For URL-based GET searches, use the exported table endpoint in step 4.

Harper applies the availability and optional category conditions while accepting current product records. Existing secondary indexes can narrow candidates; rejected records do not count toward the requested page.

Calls through tables and databases use Harper's trusted server-side context. Forwarding the resource's context preserves the authenticated caller, and checkPermission: true asks the table to enforce that caller's table and source-field permissions. The custom resource must do both. Highlights can expose searched source text even when those fields are absent from select.

Matches in tags contribute to search results and ranking, but $highlights does not include tag matches because tags is not configured with highlight: true.

4. Search through REST

The exported table supports the same match comparator:

GET /Product/?catalogSearch=matches=waterproof%20trail&category=footwear&select(id,name,price,$score,$highlights)&limit(20)

For a phrase:

GET /Product/?catalogSearch=matches_phrase=trail%20running&limit(20)

For record autocomplete:

GET /Product/?catalogSearch=matches_prefix=waterproof%20tra&select(id,name)&limit(10)

Prefix search returns matching products. It does not return a dictionary of suggested terms. Before v5.4, prefix and fuzzy-prefix queries restrict offset + limit to 100. In v5.4, bounded pages can go beyond that window within request budgets; each page runs a new query rather than retaining a search. See paging and prefix result windows.

REST searches every source field in the index, so the caller must be allowed to read name, description, and tags for these requests. Use Table.search() or search_by_conditions when a role should search only a subset of sources. See REST full-text operators.

5. Choose consistency behavior

Most searches can use the default three-second lag tolerance. When the next request must find a record already committed on the queried node, use the JavaScript API and wait for current index coverage. This wait does not wait for replication to deliver writes from another node:

for await (const product of databases.catalog.Product.search({
	conditions: [
		{
			attribute: 'catalogSearch',
			comparator: 'matches',
			value: 'new seasonal product',
			maxIndexLagMilliseconds: 0,
			waitForIndexMilliseconds: 10000,
		},
	],
})) {
	console.log(product);
}

Do not apply this wait to every catalog request without measuring it. The normal bounded-staleness path keeps search traffic independent of short indexing bursts.

If the index does not reach current coverage within ten seconds, iteration throws an error with statusCode: 503, code: "DERIVED_INDEX_LAGGING", and retryable: true. See freshness controls.

6. Operate the index

Use describe_table to inspect readiness. Harper reuses compatible local index files after restart and replays changes after their checkpoint. A node rebuilds locally when the files are missing, incompatible, or corrupt; ordinary table reads remain available, while full-text requests in unknown, needs-rebuild, or rebuilding state return 503 with code: "INDEX_REBUILDING" and retryable: true. Retry this code with backoff; do not treat every 503 as an index rebuild. Writes can also receive retryable DERIVED_INDEX_LAGGING responses while replay catches up after a long rebuild; see write backpressure.

The custom resource in step 3 streams its returned iterable. HTTP error statuses apply before response headers are sent; a failure after streaming starts is a terminal stream error and cannot change the HTTP status. Consumers must also handle that terminal error. See freshness and streaming errors.

See the full-text configuration reference and query reference for every option and match mode.

What you will build​

Prerequisites​

1. Declare the index​

2. Add records​

3. Search from a Harper application​

4. Search through REST​

5. Choose consistency behavior​

6. Operate the index​

Additional resources​