Full-Text Search Configuration

Added in: v5.3.0 Engine: RocksDB

Declare a nullable FullText field on an audited RocksDB table and attach @fullText to that field. The field name identifies the index. Add another FullText field when one table needs independent indexes for different search experiences.

type Product @table(database: "catalog", audit: true) {
	id: ID @primaryKey
	name: String
	description: String
	tags: [String]
	category: String @indexed
	price: Float @indexed
	available: Boolean
	catalogSearch: FullText
		@fullText(
			fields: [
				{ name: "name", weight: 3, highlight: true }
				{ name: "description", weight: 1, highlight: true }
				{ name: "tags", weight: 1 }
			]
			highlighting: { maxFragments: 2, fragmentLength: 120 }
		)
}

catalogSearch is a query-only declaration. It is not stored, selectable, writable, or included in record, OpenAPI, or MCP schemas. Use the field name as the condition attribute and select $score or $highlights for search metadata.

The declaration must use nullable FullText exactly and carry one @fullText directive. Lists, non-null forms, a bare FullText field, and additional directives such as @computed, @indexed, or @allow are rejected. @fullText is not valid on a table type or any other field type.

Ordinary tables inherit logging.auditLog, which defaults to true. New tables with @fullText instead require an explicit @table(audit: true); omitting it or setting audit: false returns 400, even when the global default is enabled. This table-level requirement makes transaction logging an explicit recovery prerequisite for the derived index rather than relying on the ambient default. An existing table that already has persisted audit logging can add the field without restating audit. A previously unaudited table can enable audit: true in the same schema change that adds its first full-text field; the initial build scans current records and does not require older transaction history. Disabling audit logging while a full-text field exists is rejected.

Writing the declaration-only field or selecting it as record data returns an error, including on unsealed tables. A read-through source response containing that name fails with 502 and is not cached, indexed, or audited. For HTTP-sourced caching tables, avoid index names that collide with response wrapper fields such as data, headers, body, and status.

Directive options

Only the directive options listed below are accepted; an unknown option returns 400.

Option	Default	Description
fields	required	One or more source fields.
analyzer	"english@2"	Versioned text analyzer. english@2 is the only supported value.
stopWords	true	Removes common English terms during analysis.
positions	true	Stores token positions. Required for phrase queries.
surfaceTerms	true	Stores unstemmed analyzed terms. Required for prefix, fuzzy-prefix, and highlighting.
synonyms	[]	Index-time synonym expansion rules.
highlighting	omitted	Enables highlight generation and sets fragment limits.

Each field entry supports:

Option	Default	Description
name	required	Stored source field name.
weight	1	Positive relevance weight. Higher values contribute more to BM25 ranking.
highlight	false	Allows highlights from this field when index highlighting is enabled.
mediaType	omitted	Required as "text/plain" for a Blob source; invalid on other source types.

Source fields cannot use @computed or @relationship. Indexable source values are strings, arrays containing strings or nulls, or Blob values declared as plain text.

Structured filters

Combine full-text conditions with ordinary Harper conditions using and. Harper checks current source records and can use existing secondary indexes to narrow candidates. See combining conditions.

Phrase and prefix support

positions and surfaceTerms trade index size for query features. Keep their defaults unless storage measurements show that the features are unnecessary.

type Article @table(audit: true) {
	id: ID @primaryKey
	body: String
	bodySearch: FullText @fullText(fields: [{ name: "body" }], positions: false, surfaceTerms: false)
}

This smaller declaration supports matches, matches_all, and matches_fuzzy. It rejects phrase, prefix, fuzzy-prefix, and highlighting requests.

Highlighting

Highlighting is off by default. To enable it:

1. Set highlight: true on at least one source field.
2. Add highlighting to the directive.
3. Keep surfaceTerms: true.

articleSearch: FullText
	@fullText(
		fields: [{ name: "title", highlight: true }, { name: "body", highlight: true }]
		highlighting: { maxFragments: 3, fragmentLength: 160 }
	)

maxFragments and fragmentLength must be positive integers. Query results contain offsets into the original field value; see Highlights.

Synonyms

Synonyms expand terms while records are indexed:

catalogSearch: FullText
	@fullText(
		fields: [{ name: "description" }]
		synonyms: [
			{ source: "sneaker", replacements: ["shoe", "trainer"] }
			{ source: "tv", replacements: ["television"] }
		]
	)

Rules require one source term and one or more unique replacement terms. A rule cannot replace a term with itself.

Multiple indexes

Add separate FullText fields when separate index definitions are useful:

type Product @table(database: "catalog", audit: true) {
	id: ID @primaryKey
	name: String
	tags: [String]
	titleSearch: FullText @fullText(fields: [{ name: "name", weight: 2 }])
	tagSearch: FullText @fullText(fields: [{ name: "tags" }])
}

The indexes update independently and can be queried independently. A single query cannot combine conditions from different full-text indexes.

Blob sources

Only plain-text Blob fields are supported:

type Document @table(audit: true) {
	id: ID @primaryKey
	content: Blob
	contentSearch: FullText @fullText(fields: [{ name: "content", mediaType: "text/plain" }])
}

mediaType: "text/plain" declares the expected Blob media type. Harper does not reject the source record write, but the derived index checks a nonempty runtime Blob.type and excludes the record when it does not match text/plain. The index decodes matching Blobs as UTF-8. A Blob source is limited to 1 MiB (1,048,576 bytes). A media-type mismatch, oversized value, or invalid UTF-8 removes the whole record from the index rather than indexing only its other fields. A transient Blob read failure rolls back the accepted native batch and retries it, so index coverage does not advance past unread source data.

Schema changes and rebuilds

The english@2 name remains unchanged in v5.4, but Tantivy 0.27 changes some English stems. Existing indexes must rebuild, and matching or ranking can change. See upgrading existing indexes.

Changes that affect indexed storage create a new local index generation and rebuild from source data. These include source membership, source order, media types, analyzer settings, stop-word handling, positions, surface terms, and synonyms. Keep the fields list in a stable order when its meaning has not changed.

While a generation rebuilds, ordinary table reads remain available, but full-text queries on that node return retryable 503 responses until the index is ready. Writes can receive retryable DERIVED_INDEX_LAGGING responses while replay catches up after a long rebuild; see write backpressure.

Changing source weights or highlighting settings changes query behavior without rebuilding indexed term storage. Removing the FullText field retires its local files after the schema change is confirmed. Renaming the field creates a new index identity and retires the old one.

Inspecting an index

With readable index sources, describe_table reports full_text_indexes, including fields, analyzer options, supported query modes, and readiness:

{
	"operation": "describe_table",
	"database": "catalog",
	"table": "Product"
}

Metadata respects the caller's attribute permissions: unreadable source fields are omitted, and an index with no readable source fields is omitted entirely. An omitted entry does not by itself mean that the index is unconfigured. With access to its source fields, the table response includes entries shaped like this:

{
	"full_text_indexes": [
		{
			"name": "catalogSearch",
			"fields": [{ "name": "name", "weight": 3, "highlight": true }],
			"analyzer": "english@2",
			"stop_words": true,
			"positions": true,
			"surface_terms": true,
			"synonyms": [],
			"highlighting": { "maxFragments": 2, "fragmentLength": 120 },
			"query_modes": ["any", "all", "fuzzy", "phrase", "prefix", "fuzzy-prefix"],
			"readiness": { "state": "ready", "owner_epoch": "4", "rebuild_attempts": 0 }
		}
	]
}

readiness.state is unknown, ready, needs-rebuild, rebuilding, or unavailable. Only ready serves full-text queries. unknown is an activation or owner-election transition. needs-rebuild means the runtime has condemned the current generation and queued an automatic rebuild. Queries in unknown, needs-rebuild, or rebuilding return 503 with retryable code INDEX_REBUILDING. unavailable means activation failed or the automatic rebuild budget was exhausted; queries return a generic, non-retryable 503. Inspect the optional readiness.reason and the node logs, correct the underlying storage, native-module, schema, or transaction-log problem, then reload or reapply the schema to retry activation.

query_modes maps to comparators as follows: any → matches, all → matches_all, fuzzy → matches_fuzzy, phrase → matches_phrase, prefix → matches_prefix, and fuzzy-prefix → matches_fuzzy_prefix. owner_epoch is an opaque decimal string used to fence work between derived-index owners; clients should not parse or persist it. rebuild_attempts is the current automatic-attempt count. reason is present when readiness has an operator-actionable explanation.

The Operations API uses snake_case for response metadata such as stop_words, surface_terms, and owner_epoch. The nested highlighting value mirrors the GraphQL option names maxFragments and fragmentLength.