feat: add unified search engine — HNSW heap-opt, segment index, boolean/phrase/ngram/facet
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled
Clients CI / build-server (push) Has been cancelled
Clients CI / test-python (push) Has been cancelled
Clients CI / test-javascript (push) Has been cancelled
Clients CI / test-nim (push) Has been cancelled
Clients CI / test-rust (push) Has been cancelled
New src/barabadb/search/ module with 9 components: - priority_queue.nim: BoundedHeap for O(log n) search - hnsw_opt.nim: heap-based searchLayer (2.4x faster, 92-99% recall@10) - inverted.nim: segment-based index with soft-delete and compaction - phrase.nim: positional phrase + proximity search - boolean.nim: recursive descent parser (AND/OR/NOT/ranges/wildcards) - ngram.nim: trigram index for O(1) fuzzy/prefix/wildcard - stemmer.nim: Porter2 stemmers (EN/BG/DE/FR/RU) - facet.nim: faceted search with filter pushdown - engine.nim: UnifiedSearchEngine combining all search types Performance (dim=128, efConstruction=200): N=1K: 0.30ms search, 99.6% recall@10 N=10K: 1.09ms search, 92.6% recall@10 N=50K: 2.26ms search, 75.5% recall@10 Includes search benchmarks (benchmarks/search_bench.nim), updated docs (en/bg fts.md, en/bg search.md), and crossmodal engine integration.
This commit is contained in:
+134
-3
@@ -49,8 +49,13 @@ let tfidf = idx.searchTfidf("query terms")
|
||||
| Fuzzy search | Levenshtein distance tolerance |
|
||||
| Wildcard | Prefix, suffix, and infix wildcards |
|
||||
| Regex | Regular expression patterns |
|
||||
| Phrase search | Exact phrase matching |
|
||||
| Boolean | AND, OR, NOT operators |
|
||||
| Phrase search | Exact phrase matching with slop support |
|
||||
| Proximity search | Terms within a configurable distance window |
|
||||
| Boolean | AND, OR, NOT operators with nested expressions |
|
||||
| Faceted search | Category filtering, counts, and aggregation |
|
||||
| Hybrid search | Combined full-text + vector (HNSW) with RRF fusion |
|
||||
| Segment indexing | Incremental indexing with automatic compaction |
|
||||
| Field boosting | Per-field relevance weights |
|
||||
|
||||
## SQL Interface
|
||||
|
||||
@@ -84,4 +89,130 @@ Features per language:
|
||||
- Tokenization
|
||||
- Stop words
|
||||
- Stemming
|
||||
- Language detection
|
||||
- Language detection
|
||||
|
||||
## Advanced Search
|
||||
|
||||
The new `src/barabadb/search/` module provides a unified search engine with segment-based indexing for high-performance search operations.
|
||||
|
||||
### UnifiedSearchEngine
|
||||
|
||||
```nim
|
||||
import barabadb/search/engine
|
||||
|
||||
# Create search engine with default configuration
|
||||
var engine = newUnifiedSearchEngine()
|
||||
|
||||
# Index documents with fields and facets
|
||||
engine.indexDocument(
|
||||
docId = 1,
|
||||
text = "Nim is a fast programming language",
|
||||
fields = {"title": "Nim Overview"}.toTable,
|
||||
facets = {"category": @["programming"], "level": @["beginner"]}.toTable
|
||||
)
|
||||
|
||||
# Basic search
|
||||
let results = engine.search("programming language", limit = 10)
|
||||
|
||||
# Phrase search (exact phrase matching)
|
||||
let phrase = engine.searchPhrase(@["fast", "programming"], slop = 0)
|
||||
|
||||
# Proximity search (terms within distance)
|
||||
let proximity = engine.searchProximity(@["fast", "language"], maxDistance = 5)
|
||||
|
||||
# Boolean queries
|
||||
let boolResults = engine.searchBoolean("programming AND (fast OR efficient)")
|
||||
let boolResults2 = engine.searchBoolean("Nim AND NOT Python")
|
||||
let boolResults3 = engine.searchBoolean("\"exact phrase\" OR wildcard*")
|
||||
|
||||
# Fuzzy search with typo tolerance
|
||||
let fuzzy = engine.searchFuzzy("programing", maxDistance = 2)
|
||||
|
||||
# Prefix and wildcard search
|
||||
let prefix = engine.searchPrefix("prog", limit = 10)
|
||||
let wildcard = engine.searchWildcard("prog*", limit = 10)
|
||||
```
|
||||
|
||||
### Faceted Search
|
||||
|
||||
```nim
|
||||
import barabadb/search/engine
|
||||
import std/sets
|
||||
|
||||
# Index documents with facets
|
||||
engine.indexDocument(
|
||||
docId = 1,
|
||||
text = "Nim tutorial",
|
||||
facets = {"category": @["programming", "tutorial"], "difficulty": @["beginner"]}.toTable
|
||||
)
|
||||
|
||||
# Get facet counts
|
||||
let counts = engine.getFacetCounts("category", limit = 10)
|
||||
for count in counts:
|
||||
echo count.value, ": ", count.count
|
||||
|
||||
# Filter by facets
|
||||
var filters = @[
|
||||
FacetFilter(field: "category", values: @["programming"], exclude: false),
|
||||
FacetFilter(field: "difficulty", values: @["advanced"], exclude: true)
|
||||
]
|
||||
let matchingDocs = engine.filterByFacets(filters)
|
||||
|
||||
# Aggregate multiple facets
|
||||
let agg = engine.facets.aggregate(@["category", "difficulty"], matchingDocs)
|
||||
```
|
||||
|
||||
### Hybrid Search (Text + Vector)
|
||||
|
||||
```nim
|
||||
import barabadb/search/engine
|
||||
import barabadb/vector/engine
|
||||
|
||||
# Index vectors
|
||||
engine.indexVector(1, @[0.1, 0.2, 0.3], {"title": "Doc 1"}.toTable)
|
||||
|
||||
# Hybrid search combining text and vector similarity
|
||||
let hybrid = engine.hybridSearch(
|
||||
queryText = "programming",
|
||||
queryVec = @[0.1, 0.2, 0.3],
|
||||
k = 10,
|
||||
textWeight = 1.0,
|
||||
vecWeight = 1.0
|
||||
)
|
||||
|
||||
# Filtered vector search
|
||||
proc filterMeta(meta: Table[string, string]): bool =
|
||||
meta.getOrDefault("category") == "programming"
|
||||
|
||||
let filtered = engine.searchVectorFiltered(@[0.1, 0.2, 0.3], k = 10, filterMeta)
|
||||
```
|
||||
|
||||
### Configuration and Management
|
||||
|
||||
```nim
|
||||
# Custom configuration
|
||||
var config = defaultSearchConfig()
|
||||
config.language = langBulgarian
|
||||
config.maxSegmentSize = 100_000
|
||||
config.ngramSize = 3
|
||||
config.enableFacets = true
|
||||
|
||||
var engine = newUnifiedSearchEngine(config)
|
||||
|
||||
# Set field boosts for relevance tuning
|
||||
engine.setFieldBoost("title", 2.0)
|
||||
engine.setFieldBoost("body", 1.0)
|
||||
|
||||
# Change language
|
||||
engine.setLanguage(langBulgarian)
|
||||
|
||||
# Compact segments for better performance
|
||||
engine.compact()
|
||||
|
||||
# Get statistics
|
||||
echo "Documents: ", engine.documentCount()
|
||||
echo "Terms: ", engine.termCount()
|
||||
|
||||
# Remove documents
|
||||
engine.removeDocument(1)
|
||||
```
|
||||
Reference in New Issue
Block a user