docs(readme): add AI-Native Data Platform features (Sessions 10-12)
CI / test (push) Has been cancelled
CI / verify (push) Has been cancelled

- Update Why BaraDB comparison table with Hybrid RAG, AI Agents,
  MCP Server, LangChain integration, Graph SQL, Cypher
- Add Hybrid RAG Search section under Vector Engine
- Add Graph SQL Integration section with CREATE GRAPH, GRAPH_TABLE,
  Cypher translation, and similarity/node2vec algorithms
- Add new AI-Native Data Platform section covering:
  - Natural Language → SQL (nl_to_sql, schema_prompt)
  - Text Chunking & Auto-Embedding (chunk, embed_text)
  - MCP Server (Model Context Protocol over STDIO)
  - LangChain Vector Store (Python + JS)
  - Chat Message History with RLS
- Update Project Structure with ai/ and mcp/ directories
- Update Roadmap Progress with v1.1.2 features
This commit is contained in:
2026-05-17 17:05:47 +03:00
parent 1e38e29f25
commit 3ac53ecda2
+195 -2
View File
@@ -30,8 +30,13 @@ single 3.3MB binary with no runtime dependencies.
| Core language | Python + Cython + Rust | **100% Nim** | | Core language | Python + Cython + Rust | **100% Nim** |
| Storage backend | PostgreSQL only | **Native multi-engine** | | Storage backend | PostgreSQL only | **Native multi-engine** |
| Vector search | pgvector extension | **Built-in HNSW/IVF-PQ** | | Vector search | pgvector extension | **Built-in HNSW/IVF-PQ** |
| Graph algorithms | None | **BFS, DFS, Dijkstra, PageRank, Louvain** | | Hybrid RAG search | None | **Vector + FTS + RRF reranking** |
| Graph algorithms | None | **BFS, DFS, Dijkstra, PageRank, Louvain + Cypher** |
| Graph SQL integration | None | **CREATE GRAPH, GRAPH_TABLE(), SQL-native** |
| Full-text search | PG FTS extension | **Built-in BM25 + TF-IDF** | | Full-text search | PG FTS extension | **Built-in BM25 + TF-IDF** |
| AI Agents / NL→SQL | None | **Built-in `nl_to_sql()`, `schema_prompt()`** |
| MCP Server | None | **STDIO JSON-RPC for AI tools** |
| LangChain integration | External adapters | **Native Vector Store (Python + JS)** |
| Embedded mode | No | **Yes (SQLite-like)** | | Embedded mode | No | **Yes (SQLite-like)** |
| Binary size | ~50MB+ | **3.3MB** | | Binary size | ~50MB+ | **3.3MB** |
| Dependencies | PostgreSQL, Python, many libs | **Zero** | | Dependencies | PostgreSQL, Python, many libs | **Zero** |
@@ -328,6 +333,29 @@ Features:
- **Quantization** — scalar 8-bit/4-bit, product, binary - **Quantization** — scalar 8-bit/4-bit, product, binary
- **Metadata filtering** — filter results by key-value pairs - **Metadata filtering** — filter results by key-value pairs
### Hybrid RAG Search
Combine vector similarity with full-text search and reciprocal rank fusion (RRF):
```sql
-- Hybrid search: vector + FTS reranked with RRF
SELECT hybrid_search('articles', 'embedding', 'body',
'machine learning', '[0.1, 0.2, ...]', 10) AS results;
-- With metadata pre-filtering (tenant isolation)
SELECT hybrid_search_filtered('articles', 'embedding', 'body',
'AI trends', '[0.1, 0.2, ...]', 10,
'tenant_id', 'company-a') AS results;
-- Re-rank existing results
SELECT rerank('machine learning', '[{"id":"1","score":"0.9"}, ...]') AS boosted;
```
Features:
- **Reciprocal Rank Fusion** — merges HNSW vector and BM25 FTS rankings
- **Metadata pre-filtering** — HNSW search with relational column filters
- **SQL functions** — `hybrid_search()`, `hybrid_search_ids()`, `hybrid_search_filtered()`, `rerank()`
### Graph Engine ### Graph Engine
Adjacency list storage with built-in algorithms. Adjacency list storage with built-in algorithms.
@@ -353,6 +381,158 @@ Algorithms:
- **PageRank** — node importance ranking - **PageRank** — node importance ranking
- **Louvain** — community detection - **Louvain** — community detection
- **Pattern matching** — subgraph isomorphism search - **Pattern matching** — subgraph isomorphism search
- **Similarity** — Jaccard / Adamic-Adar node similarity
- **node2vec** — random-walk graph embeddings
### Graph SQL Integration
Graph data is queryable directly through BaraQL with `CREATE GRAPH`, `GRAPH_TABLE()`, and Cypher translation:
```sql
-- Create a native graph
CREATE GRAPH social_network;
-- Query via GRAPH_TABLE with algorithms
SELECT * FROM GRAPH_TABLE(
social_network,
MATCH (u:User)-[:KNOWS]->(f:User)
ALGORITHM BFS
START u.id = 1
MAXDEPTH 3
);
-- Translate Cypher to BaraQL SQL
SELECT cypher('MATCH (u:User)-[:KNOWS]->(f) RETURN f.name') AS result;
```
Features:
- **Native graph DDL** — `CREATE GRAPH` / `DROP GRAPH`
- **SQL GRAPH_TABLE** — `MATCH`, `ALGORITHM`, `START`, `END`, `MAXDEPTH`
- **Auto-sync** — INSERT into `_nodes` / `_edges` syncs adjacency lists
- **Cypher layer** — `cypher()` SQL function translates `MATCH...RETURN` to BaraQL
## AI-Native Data Platform
BaraDB is the first database engine with built-in AI primitives — not bolted-on, but native to the query engine. RAG pipelines, LLM integration, and AI agent tools run inside the database with full multi-tenant RLS isolation.
### Natural Language → SQL
Ask questions in plain English (or any language) and get executable BaraQL:
```sql
-- Generate SQL from natural language
SELECT nl_to_sql('Show me the top 5 customers by total orders') AS query;
-- Schema-aware prompt for LLM context
SELECT schema_prompt('orders') AS context;
```
Features:
- **Schema-aware** — includes table definitions, indexes, RLS policies in the prompt
- **Validation layer** — wraps generated SQL in `LIMIT 0` to verify syntax before returning
- **Self-correction** — on error, feeds the error back to the LLM for an automatic fix
- **Tenant-aware** — respects `app.tenant_id` session variables
- **OpenAI + Ollama** — configurable via `BARADB_LLM_ENDPOINT`, `BARADB_LLM_MODEL`, `BARADB_LLM_API_KEY`
### Text Chunking & Auto-Embedding
Built-in text chunking and embedding generation for RAG pipelines:
```sql
-- Chunk text into overlapping pieces
SELECT chunk(long_article, 1024, 128) AS chunks;
-- Generate embeddings via external API (OpenAI / Ollama)
SELECT embed_text('Hello world') AS vector;
```
Features:
- **chunk() SQL function** — recursive splitting by paragraph, sentence, or fixed size
- **embed_text() SQL function** — HTTP embedding client with configurable endpoint
- **Auto-embedding on INSERT** — when a `VECTOR` column is NULL but `TEXT` is present, embeddings generate automatically
- **Configurable** via `BARADB_EMBED_ENDPOINT`, `BARADB_EMBED_MODEL`, `BARADB_EMBED_API_KEY`
### MCP Server (Model Context Protocol)
BaraDB exposes an MCP server over STDIO for AI agent integration:
```bash
./build/baramcp
```
Tools available to AI agents:
- **query** — execute parameterized BaraQL with RLS isolation
- **vector_search** — semantic HNSW search with metadata filtering
- **schema_inspect** — explore tables, columns, indexes, and RLS policies
```json
{
"name": "vector_search",
"arguments": {
"table": "docs",
"column": "embedding",
"query_vector": [0.1, 0.2, ...],
"k": 10,
"tenant_id": "company-a"
}
}
```
### LangChain Integration
Native Vector Store implementations for Python and JavaScript:
**Python:**
```python
from baradb.langchain_store import BaraDBStore
store = BaraDBStore(
client=client,
table="docs",
embedding_function=OpenAIEmbeddings().embed_query,
tenant_id="company-a"
)
await store.add_texts(["hello world", "quick brown fox"])
results = await store.similarity_search("hello", k=5)
```
**JavaScript:**
```javascript
const { BaraDBStore } = require('./baradb_langchain');
const store = new BaraDBStore({
client,
table: 'docs',
embeddingFunction: async (text) => [...],
tenantId: 'company-a'
});
await store.addDocuments([{ pageContent: 'hello world' }]);
const results = await store.similaritySearch('hello', 5);
```
Features:
- **Hybrid search** — uses `hybrid_search()` / `hybrid_search_filtered()` under the hood
- **MMR reranking** — `max_marginal_relevance_search()` for diverse results
- **Multi-tenant** — respects `tenant_id` with RLS isolation
- **Metadata filters** — pre-filter vector search by relational columns
### Chat Message History
Store conversation threads in BaraDB with RLS isolation:
```python
from baradb.chat_history import BaraDBChatHistory
history = BaraDBChatHistory(
client=client,
session_id="session-123",
tenant_id="company-a",
user_id="user-42"
)
history.add_user_message("Hello, AI!")
history.add_ai_message("Hello, how can I help?")
messages = history.messages
```
### Full-Text Search ### Full-Text Search
@@ -1227,7 +1407,13 @@ src/barabadb/
├── graph/ ├── graph/
│ ├── engine.nim # Adjacency-list graph + BFS/DFS/Dijkstra/PageRank │ ├── engine.nim # Adjacency-list graph + BFS/DFS/Dijkstra/PageRank
│ ├── community.nim # Louvain community detection │ ├── community.nim # Louvain community detection
│ └── cypher.nim # Cypher-like graph query parser │ └── cypher.nim # Cypher-to-SQL translator + query parser
├── ai/
│ ├── llm.nim # LLM client for NL→SQL (OpenAI / Ollama)
│ ├── chunk.nim # Text chunking for RAG pipelines
│ └── embed.nim # HTTP embedding client (OpenAI / Ollama)
├── mcp/
│ └── server.nim # MCP STDIO server (JSON-RPC 2.0 AI tools)
├── fts/ ├── fts/
│ ├── engine.nim # Inverted index + BM25 + TF-IDF │ ├── engine.nim # Inverted index + BM25 + TF-IDF
│ └── multilang.nim # Tokenizers for EN, BG, DE, FR, RU │ └── multilang.nim # Tokenizers for EN, BG, DE, FR, RU
@@ -1278,6 +1464,13 @@ nim c -d:release -r benchmarks/bench_all.nim
| Cross-modal queries | ✅ | 100% | v1.0.0 | | Cross-modal queries | ✅ | 100% | v1.0.0 |
| Backup & Recovery | ✅ | 100% | v1.0.0 | | Backup & Recovery | ✅ | 100% | v1.0.0 |
| Client SDKs (JS, Python, Nim, Rust) | ✅ | 100% | v1.0.0 | | Client SDKs (JS, Python, Nim, Rust) | ✅ | 100% | v1.0.0 |
| Graph SQL Integration (CREATE GRAPH, GRAPH_TABLE, Cypher) | ✅ | 100% | v1.1.2 |
| Hybrid RAG Search (vector + FTS + RRF reranking) | ✅ | 100% | v1.1.2 |
| AI Chunking & Auto-Embedding (`chunk()`, `embed_text()`) | ✅ | 100% | v1.1.2 |
| NL→SQL (`nl_to_sql()`, `schema_prompt()`) | ✅ | 100% | v1.1.2 |
| MCP Server (STDIO JSON-RPC for AI agents) | ✅ | 100% | v1.1.2 |
| LangChain Vector Store (Python + JS) | ✅ | 100% | v1.1.2 |
| Production Hardening (prop tests, fuzz tests, thread safety) | ✅ | 100% | v1.1.2 |
## Current Limitations ## Current Limitations