mirror of
https://github.com/kbenestad/mdcms.git
synced 2026-06-18 15:24:32 +00:00
314 lines
No EOL
145 KiB
JSON
314 lines
No EOL
145 KiB
JSON
[
|
|
{
|
|
"file": "pages/architecture.md",
|
|
"title": "Architecture",
|
|
"section-id": "overview",
|
|
"keywords": "architecture, storage engine, query planner, replication, WAL, HNSW",
|
|
"description": "NeuralDB internal architecture \u2014 storage engine, query planner, and replication",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Architecture\n\n\n\nNeuralDB is built on a custom storage engine that co-locates relational and vector data, with a query planner that understands both SQL predicates and vector similarity operations natively.\n\n## High-Level Architecture\n\n```\nClient (psql / SDK / REST)\n \u2502\n \u25bc\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 Connection Layer \u2502\n\u2502 (PostgreSQL Wire Protocol compatible) \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Query Parser \u2502\n \u2502 (SQL + NQL ext.) \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Semantic Planner \u2502\u25c4\u2500\u2500 Statistics + Index Metadata\n \u2502 (hybrid cost model) \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Execution Engine \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502 SQL Executor \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502 ANN Executor \u2502 \u2502\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 Storage Engine \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502 Row Store \u2502 \u2502\u25c4\u2500\u2500 SST Files (columnar)\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502 Vector Store \u2502 \u2502\u25c4\u2500\u2500 HNSW Graph Files\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2502 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n \u2502 \u2502 WAL \u2502 \u2502\u25c4\u2500\u2500 Write-Ahead Log\n \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n\n## Storage Engine\n\n### Row Store\n\nNeuralDB's row store uses a Log-Structured Merge-tree (LSM) architecture inspired by RocksDB. Data is written to an in-memory write buffer (MemTable), which is periodically flushed to sorted string tables (SSTables) on disk. Background compaction merges SSTables and reclaims space.\n\nKey properties:\n- **Write-optimised**: writes are sequential, not random \u2014 excellent NVMe utilisation\n- **Columnar format**: SSTables store data column-by-column for fast analytical scans\n- **Compression**: LZ4 by default, Zstd for archival storage \u2014 typically 3\u20136\u00d7 compression ratio\n\n### Vector Store\n\nVectors are stored separately from rows in a Vector Store. The Vector Store maintains:\n\n1. **Raw vector data** \u2014 the float32 arrays, stored in compressed pages\n2. **HNSW graph** \u2014 the in-memory navigation graph for ANN search\n\nThe HNSW graph is loaded into memory on startup and kept warm. Memory required \u2248 `num_vectors \u00d7 dimensions \u00d7 4 bytes \u00d7 1.3` (1.3\u00d7 overhead for the graph structure).\n\nFor a 10M-row table with 1536-dimensional embeddings: `10M \u00d7 1536 \u00d7 4 \u00d7 1.3 \u2248 80 GB`. Plan memory accordingly.\n\n### Write-Ahead Log (WAL)\n\nAll writes (row and vector) are first written to the WAL before being applied to the storage engine. The WAL provides:\n\n- **Durability**: committed transactions survive crashes\n- **Replication**: replicas apply the WAL stream from the primary\n- **Point-in-time recovery (PITR)**: archive the WAL to recover to any point in time\n\nWAL segments are 128 MB by default and are archived to the configured storage backend (local disk, S3, GCS) upon rotation.\n\n## Query Planner\n\nThe Semantic Planner extends a PostgreSQL-compatible query planner with understanding of vector operations.\n\n### Hybrid Cost Model\n\nFor hybrid queries (vector + relational), the planner considers two physical plans:\n\n**Plan A: Pre-filter**\n```\nFilter(price < 100) \u2192 ANN(embedding, k=10)\n```\nCost: selectivity \u00d7 full_scan_cost + ANN_cost(filtered_set)\n\n**Plan B: Post-filter**\n```\nANN(embedding, k=10\u00d7estimated_filter_ratio) \u2192 Filter(price < 100)\n```\nCost: ANN_cost(full_index) + filter_cost\n\nThe planner uses column statistics (histogram, null fraction, distinct values) and vector index parameters to estimate costs. It picks the plan with the lower estimated cost.\n\n### Index Types\n\nNeuralDB supports the following index types:\n\n| Index | Data | Purpose |\n|-------|------|---------|\n| B-tree | Scalar columns | Equality, range queries |\n| Hash | Scalar columns | Equality only (faster than B-tree) |\n| GIN | JSON, arrays | Containment queries |\n| HNSW | VECTOR columns | Approximate nearest neighbour |\n| IVF-Flat | VECTOR columns | High-recall exact-ish search |\n| BRIN | Timestamp columns | Range scans on append-only data |\n\n## Replication\n\nNeuralDB uses streaming replication. The primary continuously ships WAL segments to replicas, which apply them in order.\n\n### Synchronous vs Asynchronous Replication\n\n```sql\n-- Set replication mode per-transaction\nSET synchronous_commit = 'on'; -- wait for WAL to reach all sync replicas (safest)\nSET synchronous_commit = 'local'; -- wait for local WAL flush only (faster)\nSET synchronous_commit = 'off'; -- don't wait (fastest, small durability window)\n```\n\n### Read Replicas\n\nReplicas accept `SELECT` queries. Direct read-heavy workloads to replicas:\n\n```\np"
|
|
},
|
|
{
|
|
"file": "pages/comparison.md",
|
|
"title": "Comparison",
|
|
"section-id": "overview",
|
|
"keywords": "comparison, Postgres, pgvector, Pinecone, Weaviate, MongoDB, alternatives",
|
|
"description": "How NeuralDB compares to Postgres+pgvector, Pinecone, Weaviate, and MongoDB Atlas Vector Search",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Comparison\n\nThis page provides an honest comparison of NeuralDB against the most common alternatives for applications that need vector search.\n\n## NeuralDB vs PostgreSQL + pgvector\n\npgvector is the most popular way to add vector search to an existing PostgreSQL deployment. If you already run Postgres, the low barrier to entry is attractive. Here is where the two diverge:\n\n| Feature | NeuralDB | Postgres + pgvector |\n|---------|---------|---------------------|\n| Vector index algorithm | HNSW (native) | HNSW, IVFFlat |\n| Max dimensions | 65,536 | 16,000 |\n| Index build speed | Native Rust (fast) | C extension (moderate) |\n| Parallel index builds | Yes | Limited |\n| Vector data memory isolation | Dedicated vector buffer pool | Shared with row pages |\n| Horizontal sharding | Built-in | Manual (Citus, Patroni) |\n| Automatic embeddings | Yes | No |\n| Multi-modal vectors | Yes | No (one VECTOR column type) |\n| Streaming ingestion | Yes | No |\n| Read replica for vector queries | Automatic routing | Manual |\n\n**When to choose Postgres + pgvector:** You already have a Postgres deployment, your dataset is under 10M vectors, and you do not need automatic embeddings or horizontal scaling. The operational overhead of a new database system is not worth it.\n\n**When to choose NeuralDB:** Your vector dataset exceeds 10M rows, you need horizontal sharding, you want automatic embedding pipelines, or you are starting a new project and want a purpose-built system.\n\n## NeuralDB vs Pinecone\n\nPinecone is a fully managed, purpose-built vector database. It excels at pure vector search at massive scale.\n\n| Feature | NeuralDB | Pinecone |\n|---------|---------|---------|\n| Relational data | Full SQL | Metadata filters only |\n| Hybrid queries | Single query, query planner | Metadata post-filter |\n| ACID transactions | Yes | No |\n| SQL interface | Yes | Proprietary API |\n| Self-hosted option | Yes | No |\n| Pricing model | Infrastructure cost | Per-request + storage |\n| Latency (p99, 1M vectors) | ~5ms | ~10ms (managed) |\n| Data gravity | Stays in your infra | Vendor-managed |\n\n**When to choose Pinecone:** You need a fully managed solution with no operational overhead, your workload is pure vector search with simple metadata filtering, and you are comfortable with a vendor-specific API and pricing model.\n\n**When to choose NeuralDB:** You need relational data co-located with vectors, ACID transactions, SQL compatibility, self-hosting, or lower total cost of ownership at scale.\n\n## NeuralDB vs Weaviate\n\nWeaviate is an open-source vector database with a GraphQL-based query language and built-in module support for embedding generation.\n\n| Feature | NeuralDB | Weaviate |\n|---------|---------|---------|\n| Query language | SQL (NQL) | GraphQL |\n| Relational joins | Yes | No |\n| ACID transactions | Yes | Eventually consistent |\n| SQL wire compatibility | PostgreSQL wire protocol | Proprietary |\n| Embedding modules | Yes | Yes (vectorizers) |\n| BM25 hybrid search | Yes | Yes |\n| Multi-tenancy | Row-level, schema-level | Class-level |\n| Replication | Sync + async | Eventual |\n\n**When to choose Weaviate:** You want an open-source solution with a rich ecosystem of vectorizer modules and a GraphQL interface. If your team is more comfortable with graph-shaped queries than SQL, Weaviate is a natural fit.\n\n**When to choose NeuralDB:** You need SQL, transactional guarantees, relational joins between your vector data and other structured data, or PostgreSQL wire protocol compatibility (so existing tools like dbt, Metabase, and psql work out of the box).\n\n## NeuralDB vs MongoDB Atlas Vector Search\n\nMongoDB Atlas added vector search as an extension to its document model. It is a convenient choice if you already run Atlas.\n\n| Feature | NeuralDB | MongoDB Atlas Vector Search |\n|---------|---------|---------------------------|\n| Data model | Relational + vector | Document + vector |\n| Query language | SQL | MQL (MongoDB Query Language) |\n| ACID transactions | Yes (all operations) | Yes (within a session) |\n| Horizontal scaling | Native sharding | Atlas sharding |\n| Vector index type | HNSW | ENN (exact), HNSW |\n| Full-text + vector hybrid | Yes | Yes (Atlas Search) |\n| Self-hosted | Yes | Atlas only |\n\n**When to choose MongoDB Atlas Vector Search:** Your application already uses MongoDB and you want to add vector search without changing your data model or infrastructure. The document model maps well to semi-structured data.\n\n**When to choose NeuralDB:** You need relational data integrity, SQL, lower query latency, or the ability to self-host. If your data is inherently tabular (rather than document-shaped), NeuralDB's relational model will be a better fit.\n\n## Performance Benchmarks\n\nThe following benchmarks were run against 10M 1536-dimensional vectors on equivalent hardware (32 vCPU, 128 GB RAM, NVMe SSD):\n\n| System | QPS (recall@95%) | p50 latency | p99 latency | Index build time |\n|--------|-----------------|-------------|-------------|-----------------|\n|"
|
|
},
|
|
{
|
|
"file": "pages/concepts.md",
|
|
"title": "Core Concepts",
|
|
"section-id": "overview",
|
|
"keywords": "concepts, vectors, embeddings, hybrid queries, nodes, HNSW, ANN",
|
|
"description": "Core NeuralDB concepts \u2014 vectors, embeddings, hybrid queries, and the node model",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Core Concepts\n\nUnderstanding these fundamental concepts will help you use NeuralDB effectively and make good architectural decisions for your application.\n\n## Vectors and Embeddings\n\nA **vector** is an ordered list of floating-point numbers \u2014 a point in high-dimensional space. In NeuralDB, vectors are used to represent the semantic meaning of data.\n\nAn **embedding** is a vector produced by a machine learning model that encodes the semantic meaning of its input. Similar inputs produce vectors that are close together in the embedding space. For example, the sentences \"I love dogs\" and \"I adore canines\" will produce embeddings that are close to each other, even though they share no words.\n\nNeuralDB stores embeddings as `VECTOR(n)` columns, where `n` is the dimensionality (the number of float32 values). Common dimensionalities:\n\n| Model | Dimensions |\n|-------|-----------|\n| OpenAI text-embedding-3-small | 1536 |\n| OpenAI text-embedding-3-large | 3072 |\n| Cohere embed-english-v3.0 | 1024 |\n| Google text-embedding-004 | 768 |\n| BAAI/bge-m3 | 1024 |\n\n## Distance Metrics\n\nNeuralDB computes similarity between two vectors using one of three distance metrics:\n\n### Cosine Similarity\n\nMeasures the angle between two vectors. Ranges from -1 (opposite) to 1 (identical). Ideal for text embeddings produced by models trained with cosine objectives:\n\n```\ncosine_similarity(a, b) = (a \u00b7 b) / (|a| \u00d7 |b|)\n```\n\nIn NQL, use the `<=>` operator or `COSINE_SIMILARITY()` function.\n\n### Dot Product\n\nMeasures the product of vector magnitudes and the angle between them. Used when vectors are not normalised and magnitude carries information (e.g., collaborative filtering):\n\n```\ndot_product(a, b) = \u03a3(a\u1d62 \u00d7 b\u1d62)\n```\n\nIn NQL, use the `<#>` operator or `DOT_PRODUCT()` function.\n\n### Euclidean Distance (L2)\n\nMeasures straight-line distance between two points. Lower is more similar. Useful for spatial data and image embeddings:\n\n```\nl2_distance(a, b) = \u221a(\u03a3(a\u1d62 - b\u1d62)\u00b2)\n```\n\nIn NQL, use the `<->` operator or `L2_DISTANCE()` function.\n\n## Vector Indexes\n\nNeuralDB builds vector indexes using the **HNSW (Hierarchical Navigable Small World)** algorithm. HNSW provides:\n\n- Sub-linear approximate nearest neighbour (ANN) search\n- Configurable trade-off between speed and recall\n- Incremental updates (no full rebuild needed when inserting)\n\n### HNSW Parameters\n\nWhen creating a vector index:\n\n```sql\nCREATE INDEX ON documents\nUSING hnsw (embedding vector_cosine_ops)\nWITH (m = 16, ef_construction = 64);\n```\n\n| Parameter | Description | Default |\n|-----------|-------------|---------|\n| `m` | Number of bi-directional links per node. Higher = better recall, more memory | 16 |\n| `ef_construction` | Size of candidate set during index construction. Higher = better quality, slower build | 64 |\n| `ef_search` | Size of candidate set at query time. Set per-query with `SET hnsw.ef_search = 100` | 40 |\n\n### Exact vs Approximate Search\n\nBy default, vector queries use the HNSW index (approximate). For exact results (slower but 100% recall), use:\n\n```sql\nSET neuraldb.vector_scan = 'exact';\nSELECT * FROM documents ORDER BY embedding <=> :query LIMIT 10;\n```\n\n## Hybrid Queries\n\nA hybrid query combines vector similarity with relational predicates in a single query plan. The NeuralDB query planner evaluates two strategies and picks the cheaper one:\n\n1. **Pre-filter then search** \u2014 apply relational filters first to reduce the candidate set, then run ANN search on the filtered set\n2. **Post-filter** \u2014 run ANN search to get top-k candidates, then apply relational filters\n\nNeuralDB automatically selects the optimal strategy based on selectivity estimates. You can hint the planner:\n\n```sql\nSELECT * FROM documents\nWHERE /*+ PREFILTER */ category = 'news'\nORDER BY embedding <=> :query\nLIMIT 10;\n```\n\n## Tables and Schemas\n\nNeuralDB is schema-based, like PostgreSQL. Everything lives inside a database \u2192 schema \u2192 table hierarchy:\n\n```sql\nCREATE DATABASE my_app;\n\\c my_app\n\nCREATE SCHEMA vectors;\nCREATE SCHEMA metadata;\n\nCREATE TABLE vectors.documents (\n id UUID PRIMARY KEY DEFAULT gen_random_uuid(),\n content TEXT NOT NULL,\n embedding VECTOR(1536),\n schema_id TEXT REFERENCES metadata.schemas(id),\n created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()\n);\n```\n\n## Nodes\n\nNeuralDB is a distributed system. A **node** is a single NeuralDB process. Nodes have one of three roles:\n\n| Role | Responsibilities |\n|------|----------------|\n| **Primary** | Accepts reads and writes, coordinates transactions |\n| **Replica** | Accepts reads, replicates writes from primary |\n| **Index** | Maintains vector indexes for shard(s), offloads ANN queries |\n\nIn a single-node deployment, one process takes all three roles.\n\n## Sharding\n\nNeuralDB shards data by `shard_key`. By default, the primary key is used as the shard key:\n\n```sql\nCREATE TABLE events (\n id UUID PRIMARY KEY,\n tenant_id UUID NOT NULL,\n event_type TEXT,\n embedding VECTOR(768)\n) SHARD BY tenant_id;\n```\n\nAll rows with the same `tenant_id` are guarante"
|
|
},
|
|
{
|
|
"file": "pages/config-auth.md",
|
|
"title": "Authentication",
|
|
"section-id": "configuration",
|
|
"keywords": "authentication, API keys, mTLS, RBAC, SSO, pg_hba, security",
|
|
"description": "Configuring NeuralDB authentication \u2014 API keys, mTLS, role-based access control, and SSO",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Authentication\n\nNeuralDB supports multiple authentication methods, from simple password auth to mutual TLS and enterprise SSO. This page explains how to configure each.\n\n## Host-Based Authentication (pg_hba.conf)\n\nThe `pg_hba.conf` file controls which clients can connect and how they must authenticate. It is evaluated top-to-bottom and the first matching rule applies.\n\n```\n# pg_hba.conf\n# FORMAT: TYPE DATABASE USER ADDRESS METHOD\n\n# Local Unix socket connections (no password needed for postgres superuser)\nlocal all neuraldb trust\nlocal all all md5\n\n# IPv4 connections from local network\nhost all all 127.0.0.1/32 scram-sha-256\nhost all all 10.0.0.0/8 scram-sha-256\n\n# Reject all other connections\nhost all all 0.0.0.0/0 reject\n```\n\nReload after changes:\n\n```sql\nSELECT pg_reload_conf();\n```\n\n## Authentication Methods\n\n| Method | Security | Use case |\n|--------|---------|---------|\n| `trust` | None | Local socket, trusted environments |\n| `md5` | Weak | Legacy compatibility |\n| `scram-sha-256` | Strong (default) | Standard password auth |\n| `cert` | Very strong | Mutual TLS authentication |\n| `ldap` | Strong | Enterprise LDAP/AD integration |\n| `radius` | Strong | RADIUS server |\n| `gss` | Strong | Kerberos |\n| `jwt` | Strong | Token-based auth |\n\n### Enabling scram-sha-256\n\n```ini\n# neuraldb.conf\npassword_encryption = scram-sha-256\n```\n\nSet passwords for users:\n\n```sql\nCREATE USER app_user WITH PASSWORD 'secure-random-password';\nALTER USER app_user PASSWORD 'new-secure-password';\n```\n\n## Role-Based Access Control (RBAC)\n\n### Creating Roles\n\n```sql\n-- Application read-only role\nCREATE ROLE app_readonly;\nGRANT CONNECT ON DATABASE myapp TO app_readonly;\nGRANT USAGE ON SCHEMA public TO app_readonly;\nGRANT SELECT ON ALL TABLES IN SCHEMA public TO app_readonly;\nALTER DEFAULT PRIVILEGES IN SCHEMA public\n GRANT SELECT ON TABLES TO app_readonly;\n\n-- Application read-write role\nCREATE ROLE app_readwrite;\nGRANT app_readonly TO app_readwrite;\nGRANT INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO app_readwrite;\nALTER DEFAULT PRIVILEGES IN SCHEMA public\n GRANT INSERT, UPDATE, DELETE ON TABLES TO app_readwrite;\n\n-- Vector operations role (needed for ANN searches)\nCREATE ROLE vector_user;\nGRANT USAGE ON SCHEMA public TO vector_user;\nGRANT SELECT ON ALL TABLES IN SCHEMA public TO vector_user;\nGRANT EXECUTE ON ALL FUNCTIONS IN SCHEMA public TO vector_user;\n\n-- Application user\nCREATE USER my_app WITH PASSWORD 'app-password';\nGRANT app_readwrite TO my_app;\nGRANT vector_user TO my_app;\n```\n\n### Row-Level Security (RLS)\n\nFor multi-tenant applications, use Row-Level Security to enforce tenant isolation at the database level:\n\n```sql\nALTER TABLE documents ENABLE ROW LEVEL SECURITY;\n\n-- Tenants can only see their own documents\nCREATE POLICY tenant_isolation ON documents\n USING (tenant_id = current_setting('app.tenant_id')::UUID);\n\n-- Admin can see everything\nCREATE POLICY admin_access ON documents\n TO admin_role\n USING (true);\n```\n\nIn your application, set the tenant context before querying:\n\n```python\nwith connection.cursor() as cursor:\n cursor.execute(\"SET app.tenant_id = %s\", [tenant_id])\n cursor.execute(\"SELECT * FROM documents ORDER BY embedding <=> %s LIMIT 10\", [embedding])\n```\n\n## API Key Authentication\n\nNeuralDB supports an API key table for token-based authentication \u2014 useful for microservices and CI pipelines that cannot use password auth:\n\n```sql\n-- Enable the API key extension\nCREATE EXTENSION neuraldb_apikeys;\n\n-- Create an API key for a service account\nSELECT neuraldb_apikeys.create_key(\n label => 'my-service',\n role => 'app_readwrite'\n);\n-- Returns: ndb_live_abcdefghij123456...\n```\n\nClients authenticate by passing the key in the connection string:\n\n```\npostgresql://apikey:ndb_live_abc123@host:5432/mydb\n```\n\nRevoke a key:\n\n```sql\nSELECT neuraldb_apikeys.revoke_key('ndb_live_abc123');\n```\n\n## Mutual TLS (mTLS)\n\nFor the highest security, require clients to present a certificate signed by your CA.\n\n### 1. Generate a CA\n\n```bash\n# Generate CA key and certificate\nopenssl genrsa -out ca.key 4096\nopenssl req -new -x509 -key ca.key -out ca.crt -days 3650 \\\n -subj \"/CN=NeuralDB CA/O=My Org\"\n```\n\n### 2. Issue a Server Certificate\n\n```bash\nopenssl genrsa -out server.key 2048\nopenssl req -new -key server.key -out server.csr \\\n -subj \"/CN=neuraldb.example.com\"\nopenssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \\\n -CAcreateserial -out server.crt -days 365\n```\n\n### 3. Configure NeuralDB\n\n```ini\n# neuraldb.conf\nssl = on\nssl_cert_file = '/etc/neuraldb/ssl/server.crt'\nssl_key_file = '/etc/neuraldb/ssl/server.key'\nssl_ca_file = '/etc/neuraldb/ssl/ca.crt'\nssl_crl_file = '/etc/neuraldb/ssl/crl.pem' # optional certificate revocation list\n```\n\n### 4. Require Client Certificates\n\n```\n# pg_hba.conf\nhostssl all all 0.0.0.0/0 cert clientcert=verify-full\n```\n\n## LDAP / Active Directory\n\n```\n# pg_hba.conf\nhost all a"
|
|
},
|
|
{
|
|
"file": "pages/config-replication.md",
|
|
"title": "Replication",
|
|
"section-id": "configuration",
|
|
"keywords": "replication, primary, replica, streaming replication, multi-region, consistency",
|
|
"description": "Configuring NeuralDB replication \u2014 primary/replica setup, multi-region, and consistency levels",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Replication\n\nNeuralDB replication is based on streaming replication: the primary continuously ships WAL records to replicas, which apply them in real time. This page explains how to set up and configure replication.\n\n## Prerequisites\n\n- The primary must have `wal_level = replica` or higher\n- `max_wal_senders` must be greater than the number of replicas\n- A replication user must exist\n\n## Setting Up a Primary\n\nConfigure `neuraldb.conf` on the primary:\n\n```ini\n# neuraldb.conf (primary)\nwal_level = replica\nmax_wal_senders = 10\nmax_replication_slots = 10\nwal_keep_size = 1GB\nhot_standby_feedback = on # prevents primary from vacuuming rows still needed by replicas\n```\n\nCreate a replication user:\n\n```sql\nCREATE USER replicator WITH REPLICATION PASSWORD 'repl-password';\n```\n\nAllow the replica to connect:\n\n```\n# pg_hba.conf (primary)\nhost replication replicator replica-ip/32 scram-sha-256\n```\n\n## Setting Up a Replica\n\nOn the replica server, use `pg_basebackup` to clone the primary:\n\n```bash\n# On the replica server\npg_basebackup \\\n --host=primary.example.com \\\n --port=5432 \\\n --username=replicator \\\n --pgdata=/var/lib/neuraldb/data \\\n --wal-method=stream \\\n --checkpoint=fast \\\n --progress \\\n --write-recovery-conf\n```\n\nThe `--write-recovery-conf` flag creates a `standby.signal` file and writes connection info to `postgresql.auto.conf`, which tells NeuralDB to start in standby mode.\n\nConfigure `neuraldb.conf` on the replica:\n\n```ini\n# neuraldb.conf (replica)\nhot_standby = on # allow read queries\nhot_standby_feedback = on # send feedback to primary\nwal_receiver_timeout = 60s\nrecovery_min_apply_delay = 0 # apply WAL immediately (increase for delayed replicas)\n```\n\nStart the replica:\n\n```bash\nsystemctl start neuraldb\n```\n\nVerify replication is working:\n\n```sql\n-- On the primary\nSELECT client_addr, state, sent_lsn, write_lsn, flush_lsn, replay_lsn,\n (sent_lsn - replay_lsn) AS replication_lag_bytes\nFROM pg_stat_replication;\n```\n\n## Replication Slots\n\nReplication slots ensure the primary retains WAL until the replica has consumed it. This prevents the replica from falling too far behind, but also prevents WAL from being cleaned up if the replica disconnects.\n\n```sql\n-- Create a replication slot\nSELECT pg_create_physical_replication_slot('replica_1');\n\n-- List slots and their lag\nSELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS lag\nFROM pg_replication_slots;\n\n-- Drop a slot (do this if a replica is permanently removed)\nSELECT pg_drop_replication_slot('replica_1');\n```\n\n**Warning:** Monitor slot lag. An inactive slot with large lag will cause unbounded WAL accumulation and can fill your disk.\n\n## Synchronous Replication\n\nBy default, replication is asynchronous \u2014 the primary does not wait for replicas to acknowledge writes. For zero data loss, configure synchronous replication:\n\n```ini\n# neuraldb.conf (primary)\nsynchronous_standby_names = 'FIRST 1 (replica1, replica2)'\n# ^ Wait for at least 1 of the listed standbys to acknowledge each commit\n```\n\nModes:\n- `FIRST n (list)` \u2014 wait for the first n standbys in the list\n- `ANY n (list)` \u2014 wait for any n standbys from the list\n- `*` \u2014 wait for all standbys\n\nPer-transaction override:\n\n```sql\nSET synchronous_commit = 'local'; -- this transaction doesn't wait for replicas\n```\n\n## Multi-Region Replication\n\nFor global deployments, replicate to remote regions. The configuration is identical to local replication, but network latency affects synchronous commit performance.\n\nRecommended approach for multi-region:\n\n```\nPrimary (us-east-1)\n\u251c\u2500\u2500 Sync replica (us-east-1-az2) \u2190 HA within region, ~2ms latency\n\u251c\u2500\u2500 Async replica (eu-west-1) \u2190 EU reads, ~80ms latency\n\u2514\u2500\u2500 Async replica (ap-northeast-1) \u2190 APAC reads, ~170ms latency\n```\n\n```ini\n# Synchronous only within region; async to remote regions\nsynchronous_standby_names = 'FIRST 1 (local_replica)'\n```\n\nConfigure the remote replicas with a `primary_conninfo` pointing to the primary:\n\n```ini\n# standby.signal (on replica)\nprimary_conninfo = 'host=primary.us-east-1.example.com port=5432 user=replicator password=repl-password sslmode=require'\n```\n\n## Failover\n\nNeuralDB does not include automatic failover out of the box. Use one of:\n\n- **Patroni** \u2014 industry-standard HA manager for PostgreSQL-compatible databases\n- **NeuralDB HA Operator** \u2014 Kubernetes operator with automatic failover (see [Kubernetes docs](install-kubernetes.md))\n- **repmgr** \u2014 lightweight failover manager\n\nManual failover:\n\n```bash\n# On the replica that will become the new primary\nneuraldb-cli -c \"SELECT pg_promote();\"\n```\n\nAfter promotion, update `primary_conninfo` on all other replicas to point to the new primary.\n\n## Monitoring Replication\n\n```sql\n-- Replication lag in bytes and seconds\nSELECT client_addr, state,\n pg_size_pretty(sent_lsn - replay_lsn) AS lag_bytes,\n now() - pg_last_xact_replay_timestamp() AS lag_time\nFROM pg_stat_replication;\n\n-- On a replica: chec"
|
|
},
|
|
{
|
|
"file": "pages/config-server.md",
|
|
"title": "Server Config",
|
|
"section-id": "configuration",
|
|
"keywords": "neuraldb.conf, server configuration, settings, parameters, tuning",
|
|
"description": "Complete reference for neuraldb.conf \u2014 all server configuration settings explained",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Server Config\n\nNeuralDB is configured through `neuraldb.conf`, a key-value text file. This page documents all configuration parameters.\n\n## Locating the Config File\n\n```bash\n# Show the active config file path\nneuraldb-cli -c \"SHOW config_file;\"\n```\n\nDefault locations:\n- Linux: `/etc/neuraldb/neuraldb.conf`\n- macOS Homebrew: `$(brew --prefix)/etc/neuraldb/neuraldb.conf`\n- Docker: `/var/lib/neuraldb/data/neuraldb.conf`\n\nChanges to `neuraldb.conf` require a reload (for most parameters) or a restart:\n\n```bash\n# Reload without restart (applies most parameters)\nneuraldb-cli -c \"SELECT pg_reload_conf();\"\n\n# Full restart (required for listen_addresses, port, shared_buffers, etc.)\nsystemctl restart neuraldb\n```\n\n## Connection Settings\n\n```ini\n# Network interface to listen on\n# '*' = all interfaces, 'localhost' = local only\nlisten_addresses = '*'\n\n# TCP port\nport = 5432\n\n# Maximum simultaneous connections\n# Each connection uses ~5 MB of memory\nmax_connections = 100\n\n# Unix domain socket directory (Linux/macOS only)\nunix_socket_directories = '/var/run/neuraldb'\n\n# Superuser reserved connections\n# Reserves this many connections exclusively for superusers\nsuperuser_reserved_connections = 3\n```\n\n## Memory Settings\n\nThese are the most impactful parameters for performance.\n\n```ini\n# Page cache for relational data (row store)\n# Recommended: 25% of available RAM\nshared_buffers = 4GB\n\n# Memory for HNSW vector indexes\n# Recommended: 40-60% of available RAM for vector-heavy workloads\n# Must be large enough to fit all active HNSW graphs\nvector_buffer = 8GB\n\n# Per-query working memory (sorts, hash joins)\n# Recommended: 64MB\u2013256MB for OLTP, more for analytical queries\n# Be conservative: (max_connections \u00d7 work_mem) should fit in RAM\nwork_mem = 128MB\n\n# Memory for DDL maintenance (CREATE INDEX, VACUUM, etc.)\nmaintenance_work_mem = 2GB\n\n# Shared memory for parallel query workers\nparallel_query_mem = 512MB\n```\n\n## Vector Settings\n\n```ini\n# Default HNSW ef_search parameter (candidates evaluated per query)\n# Higher = better recall, slower queries\nhnsw.ef_search = 40\n\n# Maximum number of vectors per shard before auto-splitting\nvector_shard_size = 10000000\n\n# Enable approximate nearest neighbour by default (true = HNSW index)\n# Set to 'exact' to always use exact search (ignores vector indexes)\nvector_scan = 'approximate'\n\n# Number of parallel threads for HNSW index builds\nhnsw.build_threads = 4\n\n# Compression algorithm for stored vector data\n# 'none', 'lz4', 'scalar_quantize' (lossy but 4\u00d7 smaller)\nvector_compression = 'lz4'\n```\n\n## WAL Settings\n\n```ini\n# WAL logging level\n# 'minimal': minimum for crash recovery\n# 'replica': enables streaming replication (default)\n# 'logical': enables logical replication and CDC\nwal_level = replica\n\n# WAL segment size (change requires initdb)\nwal_segment_size = 128MB\n\n# Maximum number of concurrent WAL sender processes\nmax_wal_senders = 10\n\n# Number of WAL segments to keep for standby catching up\nwal_keep_size = 1GB\n\n# Synchronise WAL to disk before acknowledging commit\n# 'on': safest; 'local': only local disk; 'off': async (fastest, tiny durability risk)\nsynchronous_commit = on\n\n# Interval between WAL checkpoints\ncheckpoint_timeout = 5min\ncheckpoint_completion_target = 0.9\nmax_wal_size = 4GB\n```\n\n## Replication Settings\n\n```ini\n# Comma-separated list of standby names that must acknowledge writes\n# Leave empty for asynchronous replication\nsynchronous_standby_names = ''\n\n# Maximum lag allowed before primary throttles writes\nmax_standby_lag = 30s\n\n# Hot standby: allow reads on replicas\nhot_standby = on\n```\n\n## Query Planner\n\n```ini\n# Estimated cost of a random page fetch (tune based on SSD vs HDD)\n# Lower values favour index scans; higher values favour sequential scans\nrandom_page_cost = 1.1 # NVMe SSD (default is 4.0 for HDD)\n\n# Effective size of the disk cache (affects planner estimates)\neffective_cache_size = 24GB # ~75% of RAM\n\n# Enable parallel query\nmax_parallel_workers_per_gather = 4\nmax_parallel_workers = 8\nmax_worker_processes = 16\n\n# Hybrid query planner behaviour\n# 'auto': planner decides pre-filter vs post-filter\n# 'pre-filter': always pre-filter\n# 'post-filter': always post-filter\nvector_hybrid_strategy = 'auto'\n```\n\n## Logging\n\n```ini\n# Log destination\nlog_destination = 'stderr' # 'stderr', 'csvlog', 'jsonlog', 'syslog'\n\n# Minimum severity to log\n# 'DEBUG5' (verbose) \u2192 'INFO' \u2192 'WARNING' \u2192 'ERROR' \u2192 'FATAL'\nlog_min_messages = WARNING\n\n# Log all SQL statements with duration above this threshold (ms)\n# -1 = disable; 0 = log everything; 250 = log slow queries only\nlog_min_duration_statement = 250\n\n# Log query parameters\nlog_parameters = off\n\n# Log connection events\nlog_connections = on\nlog_disconnections = on\n\n# Log lock waits longer than this (ms)\ndeadlock_timeout = 1s\nlog_lock_waits = on\n```\n\n## Full Configuration Example\n\n```ini\n# neuraldb.conf \u2014 Production settings for a 32 vCPU / 128 GB server\n\nlisten_addresses = '*'\nport = 5432\nmax_connections = 500\nsuperuser_reserved_connections"
|
|
},
|
|
{
|
|
"file": "pages/config-storage.md",
|
|
"title": "Storage Config",
|
|
"section-id": "configuration",
|
|
"keywords": "storage, memory, disk, SSD tiers, compression, tablespace, IOPS",
|
|
"description": "Configuring NeuralDB storage \u2014 memory tiers, disk layout, compression, and tablespaces",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Storage Config\n\nNeuralDB's performance depends heavily on storage configuration. This page covers how to optimise disk layout, configure memory tiers, enable compression, and use tablespaces for data tiering.\n\n## Disk Layout Recommendations\n\nSeparate the data directory, WAL, and vector files onto different physical disks (or at least different volumes with guaranteed IOPS):\n\n| Directory | Recommended storage | IOPS requirement |\n|-----------|--------------------|--------------------|\n| `$DATADIR/base/` | NVMe SSD | High random read/write |\n| `$DATADIR/wal/` | NVMe SSD (separate) | High sequential write |\n| `$DATADIR/vectors/` | NVMe SSD or RAM disk | High random read |\n| `$DATADIR/archive/` | HDD or object storage | Low (sequential write) |\n\nConfigure separate paths in `neuraldb.conf`:\n\n```ini\n# Separate WAL onto a dedicated volume\nwal_directory = '/mnt/nvme-wal/neuraldb/wal'\n\n# Separate vector storage\nvector_data_directory = '/mnt/nvme-fast/neuraldb/vectors'\n\n# Archive destination for WAL shipping\narchive_status_directory = '/var/lib/neuraldb/archive_status'\n```\n\n## Memory Tiers\n\nNeuralDB has three distinct memory pools:\n\n### shared_buffers (Row Store Cache)\n\nThe page cache for relational data. Sized at 25% of available RAM for a dedicated database server:\n\n```ini\nshared_buffers = 32GB # for a 128 GB server\n```\n\n### vector_buffer (Vector Index Cache)\n\nHolds the HNSW graph in memory. The entire active HNSW graph must fit in `vector_buffer` for optimal performance. When the graph doesn't fit, NeuralDB falls back to disk-based graph traversal, which is 10\u201350\u00d7 slower.\n\nCalculate the required size:\n\n```\nvector_buffer = num_vectors \u00d7 dimensions \u00d7 4 bytes \u00d7 hnsw_overhead_factor\n```\n\nWhere `hnsw_overhead_factor` \u2248 1.3 for default HNSW parameters (m=16).\n\n```sql\n-- Check current vector index memory usage\nSELECT table_name, index_name,\n pg_size_pretty(hnsw_graph_size_bytes) AS graph_memory,\n pg_size_pretty(vector_data_size_bytes) AS data_size\nFROM neuraldb_stat_vector_indexes;\n```\n\n### work_mem (Per-Query Buffer)\n\nUsed for in-memory sorts, hash joins, and bitmap operations. Set conservatively \u2014 each query can allocate multiple `work_mem` buffers:\n\n```ini\n# For 200 connections with typical 2 buffers per query:\n# Max memory consumption: 200 \u00d7 2 \u00d7 128MB = 51 GB\nwork_mem = 128MB\n```\n\nOverride per-session for analytical queries:\n\n```sql\nSET work_mem = '2GB';\nSELECT ... complex analytical query ...\n```\n\n## Compression\n\n### Row Store Compression\n\nNeuralDB compresses SSTables on disk. Choose the algorithm based on your priorities:\n\n```ini\n# Compression algorithm for row data\n# 'none': no compression (fastest reads/writes, most disk)\n# 'lz4': fast, moderate compression ratio (~2-3\u00d7) \u2014 default\n# 'zstd': slower compression, better ratio (~3-5\u00d7)\n# 'zstd-9': high compression for archival (slow, ~6-8\u00d7)\nstorage_compression = lz4\n\n# Compression level (for zstd only), 1-19\nstorage_compression_level = 3\n```\n\n### Vector Compression\n\n```ini\n# Vector data compression\n# 'none': full precision, largest storage\n# 'lz4': fast, minimal precision loss\n# 'scalar_quantize': reduce to 8-bit (4\u00d7 smaller, ~1% recall loss)\n# 'product_quantize': very high compression, higher recall loss\nvector_compression = lz4\n```\n\nTo enable scalar quantisation for a specific index:\n\n```sql\nCREATE INDEX ON documents\nUSING hnsw (embedding vector_cosine_ops)\nWITH (quantization = 'scalar');\n```\n\nScalar-quantised indexes use 4\u00d7 less memory and may be faster due to better cache utilisation, at a typical recall cost of 0.5\u20132%.\n\n## Tablespaces\n\nUse tablespaces to store different tables or indexes on different volumes:\n\n```sql\n-- Create tablespaces pointing to different mount points\nCREATE TABLESPACE fast_ssd LOCATION '/mnt/nvme-fast';\nCREATE TABLESPACE bulk_hdd LOCATION '/mnt/hdd-storage';\n\n-- Create a table on fast SSD\nCREATE TABLE hot_documents (\n id UUID PRIMARY KEY,\n content TEXT,\n embedding VECTOR(1536)\n) TABLESPACE fast_ssd;\n\n-- Move an index to a specific tablespace\nCREATE INDEX ON hot_documents USING hnsw (embedding vector_cosine_ops)\nTABLESPACE fast_ssd;\n\n-- Move old partitions to cheaper storage\nALTER TABLE documents_2024_q1 SET TABLESPACE bulk_hdd;\n```\n\n## Table Partitioning\n\nPartition large tables by time or tenant to improve query performance and manageability:\n\n```sql\n-- Partition documents by month\nCREATE TABLE documents (\n id UUID NOT NULL DEFAULT gen_random_uuid(),\n content TEXT,\n embedding VECTOR(1536),\n tenant_id UUID,\n created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW()\n) PARTITION BY RANGE (created_at);\n\nCREATE TABLE documents_2026_01\n PARTITION OF documents\n FOR VALUES FROM ('2026-01-01') TO ('2026-02-01')\n TABLESPACE fast_ssd;\n\nCREATE TABLE documents_2025_archive\n PARTITION OF documents\n FOR VALUES FROM ('2025-01-01') TO ('2026-01-01')\n TABLESPACE bulk_hdd;\n```\n\n## VACUUM and Autovacuum\n\nNeuralDB inherits PostgreSQL's MVCC-based VACUUM system:\n\n```ini\n# Autovacuum settings\nautovacuum = on\nautovacuum_naptime = 1m"
|
|
},
|
|
{
|
|
"file": "pages/index.md",
|
|
"title": "What is NeuralDB?",
|
|
"section-id": "overview",
|
|
"keywords": "NeuralDB, introduction, AI database, vector database, overview",
|
|
"description": "What NeuralDB is, its key use cases, and a high-level architecture overview",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# What is NeuralDB?\n\n\n\nNeuralDB is an AI-native database that unifies vector storage, semantic search, and relational data management in a single, horizontally scalable system. It is designed for applications that need to combine structured data queries with the semantic understanding that modern AI models provide.\n\nTraditional databases store and retrieve data based on exact matches and range queries. NeuralDB adds a third dimension: **semantic proximity**. You can ask \"find the 20 products most similar to this description\" at the same time as \"where inventory > 0 and price < 100\" \u2014 in a single, atomic query with full ACID guarantees.\n\n## Why NeuralDB Exists\n\nThe AI application stack of 2024\u20132026 exposed a fundamental tension in data architecture. Teams building retrieval-augmented generation (RAG) systems, recommendation engines, and semantic search needed:\n\n- A **vector database** (Pinecone, Weaviate, Qdrant) for embedding storage and similarity search\n- A **relational database** (PostgreSQL, MySQL) for structured metadata and transactions\n- A **synchronisation layer** to keep the two in sync \u2014 a constant source of bugs and operational overhead\n\nNeuralDB replaces all three with a single system. Vectors and relational data share the same storage engine, the same transaction log, and the same query planner. There is no synchronisation problem because there is no synchronisation needed.\n\n## Key Capabilities\n\n### Hybrid Vector + Relational Queries\n\n```sql\nSELECT id, name, price, SIMILARITY(embedding, :query_embedding) AS score\nFROM products\nWHERE category = 'electronics'\n AND price BETWEEN 50 AND 500\n AND stock > 0\nORDER BY score DESC\nLIMIT 10;\n```\n\nThis query uses a vector index for semantic ranking and a B-tree index for the relational filters, with the query planner choosing the optimal execution order.\n\n### Multiple Vector Index Types\n\nNeuralDB supports three similarity metrics out of the box:\n\n| Metric | Use case |\n|--------|----------|\n| Cosine similarity | Text embeddings, normalised vectors |\n| Dot product | Recommendation systems (unnormalised) |\n| Euclidean (L2) | Image embeddings, spatial data |\n\n### Full ACID Transactions\n\nUnlike most vector databases, NeuralDB provides full ACID guarantees \u2014 including transactional upserts of both the relational row and the vector embedding simultaneously.\n\n### Automatic Embedding Updates\n\nConfigure NeuralDB to call an embedding API whenever a text column changes:\n\n```sql\nALTER TABLE documents\nADD COLUMN embedding VECTOR(1536)\nGENERATED ALWAYS AS EMBEDDING OF content\nUSING openai_ada_002;\n```\n\nNeuralDB handles the embedding pipeline automatically \u2014 no application-level embedding code required.\n\n### Multi-Modal Vectors\n\nStore and query vectors from any modality \u2014 text, image, audio, or custom \u2014 in the same table:\n\n```sql\nCREATE TABLE media_items (\n id UUID PRIMARY KEY,\n title TEXT,\n text_embedding VECTOR(1536),\n image_embedding VECTOR(512),\n created_at TIMESTAMP DEFAULT NOW()\n);\n```\n\n## Use Cases\n\n**Retrieval-Augmented Generation (RAG)** \u2014 Store your document corpus with embeddings. Query the most relevant chunks in a single round-trip and inject them into your LLM prompt.\n\n**Semantic Product Search** \u2014 Replace keyword search with semantic search. Find products matching \"comfortable running shoes for flat feet\" even if those exact words don't appear in any product description.\n\n**Recommendation Engines** \u2014 Store user preference vectors alongside item vectors. Compute collaborative-filtering recommendations with a single NQL query.\n\n**Anomaly Detection** \u2014 Flag records whose vectors are distant from the cluster of \"normal\" data.\n\n**Duplicate Detection** \u2014 Find near-duplicate records across millions of rows using approximate nearest-neighbour (ANN) search.\n\n**Knowledge Graphs** \u2014 Store entity embeddings alongside relationship metadata for graph-enhanced retrieval.\n\n## NeuralDB vs Traditional Approaches\n\n| Capability | NeuralDB | Postgres + pgvector | Pinecone + Postgres |\n|-----------|---------|---------------------|---------------------|\n| Vector search | Native, HNSW | Extension, limited | Native |\n| Relational queries | Full SQL | Full SQL | None (separate DB) |\n| Hybrid queries | Single query | Single query | Application-layer join |\n| ACID transactions | Yes | Yes | Partial |\n| Horizontal sharding | Built-in | Manual (Citus) | Managed |\n| Automatic embeddings | Yes | No | No |\n| Streaming ingestion | Yes | No | Partial |\n\n## Getting Started\n\nThe fastest way to try NeuralDB is with Docker:\n\n```bash\ndocker run -p 5432:5432 -e NEURALDB_PASSWORD=mypassword neuraldb/neuraldb:latest\n```\n\nConnect with any PostgreSQL-compatible client:\n\n```bash\npsql -h localhost -U neuraldb -d neuraldb\n```\n\nThen read the [Core Concepts](concepts.md) to understand the NeuralDB data model, or jump to [NQL Basics](nql-basics.md) to start writing queries."
|
|
},
|
|
{
|
|
"file": "pages/install-cloud.md",
|
|
"title": "Cloud Managed",
|
|
"section-id": "installation",
|
|
"keywords": "cloud, managed, NeuralDB Cloud, regions, tiers, SaaS",
|
|
"description": "Setting up NeuralDB Cloud \u2014 the fully managed service with global regions and flexible tiers",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Cloud Managed\n\nNeuralDB Cloud is the fully managed version of NeuralDB. It handles provisioning, patching, backups, monitoring, and scaling \u2014 so you can focus on building your application rather than managing database infrastructure.\n\n## Getting Started\n\n### 1. Create an Account\n\nSign up at [cloud.neuraldb.io](https://cloud.neuraldb.io). You can authenticate with Google, GitHub, or an email address.\n\n### 2. Create a Cluster\n\nClick **New Cluster** and configure:\n\n- **Region**: choose the cloud region closest to your application servers\n- **Tier**: select based on your workload requirements (see tier comparison below)\n- **Storage**: initial storage allocation (can be scaled later)\n- **High Availability**: enable for production workloads\n\n### 3. Connect\n\nOnce the cluster is provisioned (typically under 3 minutes), your connection string appears in the dashboard:\n\n```\npostgresql://neuraldb:[password]@[cluster-id].cloud.neuraldb.io:5432/[database]?sslmode=require\n```\n\nUse this with any PostgreSQL-compatible driver or psql:\n\n```bash\npsql \"postgresql://neuraldb:mypassword@abc123.cloud.neuraldb.io:5432/mydb?sslmode=require\"\n```\n\n## Available Regions\n\nNeuralDB Cloud is available in the following regions:\n\n| Region | Cloud Provider | Availability |\n|--------|---------------|-------------|\n| us-east-1 (N. Virginia) | AWS | GA |\n| us-west-2 (Oregon) | AWS | GA |\n| eu-west-1 (Ireland) | AWS | GA |\n| eu-central-1 (Frankfurt) | AWS | GA |\n| ap-northeast-1 (Tokyo) | AWS | GA |\n| ap-southeast-1 (Singapore) | AWS | GA |\n| us-central1 (Iowa) | GCP | Beta |\n| europe-west4 (Netherlands) | GCP | Beta |\n| eastus (Virginia) | Azure | Beta |\n\nMulti-region replication (primary in one region, read replicas in others) is available on Business and Enterprise tiers.\n\n## Pricing Tiers\n\n### Starter\n\nFree tier for development and experimentation.\n\n| Resource | Limit |\n|---------|-------|\n| Storage | 5 GB |\n| Vector dimensions | Up to 1536 |\n| Max connections | 10 |\n| PITR | No |\n| HA | No |\n| SLA | No |\n\nThe Starter tier automatically suspends after 7 days of inactivity and resumes on the next connection (cold start: ~30 seconds).\n\n### Developer\n\n$29/month \u2014 for side projects and pre-production environments.\n\n| Resource | Limit |\n|---------|-------|\n| vCPU | 2 dedicated |\n| RAM | 8 GB |\n| Storage | 100 GB NVMe SSD |\n| Connections | 100 |\n| PITR | 7 days |\n| HA | No |\n\n### Business\n\n$199/month \u2014 for production workloads.\n\n| Resource | Limit |\n|---------|-------|\n| vCPU | 8 dedicated |\n| RAM | 32 GB |\n| Storage | 500 GB NVMe SSD (expandable) |\n| Connections | 500 |\n| PITR | 30 days |\n| HA | Yes (1 standby) |\n| Read replicas | Up to 3 |\n| SLA | 99.95% |\n\n### Business Plus\n\n$599/month \u2014 for large-scale production workloads.\n\n| Resource | Limit |\n|---------|-------|\n| vCPU | 32 dedicated |\n| RAM | 128 GB |\n| Storage | 2 TB NVMe SSD (expandable) |\n| Connections | 2,000 |\n| PITR | 30 days |\n| HA | Yes (2 standbys) |\n| Read replicas | Up to 10 |\n| Multi-region replicas | Yes |\n| SLA | 99.99% |\n\n### Enterprise\n\nCustom pricing \u2014 for mission-critical applications with specific compliance requirements.\n\n- Dedicated infrastructure (no multi-tenancy)\n- Custom SLAs\n- Private endpoints (AWS PrivateLink, GCP Private Service Connect)\n- SOC 2 Type II, HIPAA, and ISO 27001 compliance\n- Dedicated support with SLA\n\n## Connecting from Your Application\n\n### Connection Pooling\n\nNeuralDB Cloud includes PgBouncer-based connection pooling. Use the pooler endpoint for serverless and short-lived connections:\n\n```\npostgresql://neuraldb:[password]@[cluster-id]-pooler.cloud.neuraldb.io:5432/[database]\n```\n\n| Connection type | Endpoint suffix | Use for |\n|----------------|----------------|---------|\n| Direct | (none) | Long-lived connections, COPY operations |\n| Transaction pooling | `-pooler` | Serverless, short-lived connections |\n| Session pooling | `-session-pooler` | Prepared statements |\n\n### IP Allow-listing\n\nRestrict access to specific IP ranges in the **Security** tab of your cluster dashboard. Enter your application servers' CIDR ranges (e.g., `10.0.0.0/8` for private VPC, or specific public IPs).\n\n### SSL/TLS\n\nAll connections require TLS. Download the cluster CA certificate from the dashboard and verify it in your connection string:\n\n```\nsslmode=verify-full&sslrootcert=/path/to/ca.pem\n```\n\n## Monitoring and Alerting\n\nNeuralDB Cloud includes a built-in monitoring dashboard with:\n\n- Query throughput and latency percentiles (p50, p95, p99)\n- Connection count\n- Storage usage\n- Vector index size\n- Replication lag\n\nConfigure alerts for:\n- Storage > 80% full\n- Average query latency > 500ms\n- Replication lag > 30s\n- Failed connections\n\nMetrics are also available via the NeuralDB Cloud API for ingestion into your own monitoring stack (Datadog, Grafana Cloud, New Relic).\n\n## Branching (Database Branches)\n\nNeuralDB Cloud supports **branching** \u2014 create an instant copy-on-write clone of your production database for development, testing, or migrations:\n\n```bash\nneuraldb-cl"
|
|
},
|
|
{
|
|
"file": "pages/install-docker.md",
|
|
"title": "Docker Install",
|
|
"section-id": "installation",
|
|
"keywords": "Docker, install, docker run, docker-compose, volumes, container",
|
|
"description": "Installing NeuralDB using Docker \u2014 single container and docker-compose setups",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Docker Install\n\nDocker is the fastest way to run NeuralDB locally or in a single-server deployment. NeuralDB's official Docker image is published to Docker Hub as `neuraldb/neuraldb`.\n\n## Quick Start\n\nRun a single NeuralDB instance:\n\n```bash\ndocker run -d \\\n --name neuraldb \\\n -p 5432:5432 \\\n -e NEURALDB_PASSWORD=mypassword \\\n -e NEURALDB_DB=mydb \\\n -v neuraldb_data:/var/lib/neuraldb/data \\\n neuraldb/neuraldb:latest\n```\n\nConnect with psql:\n\n```bash\npsql -h localhost -p 5432 -U neuraldb -d mydb\n# Password: mypassword\n```\n\n## Environment Variables\n\n| Variable | Default | Description |\n|----------|---------|-------------|\n| `NEURALDB_PASSWORD` | required | Password for the `neuraldb` superuser |\n| `NEURALDB_USER` | `neuraldb` | Superuser username |\n| `NEURALDB_DB` | `neuraldb` | Default database name |\n| `NEURALDB_PORT` | `5432` | TCP port |\n| `NEURALDB_MAX_CONNECTIONS` | `100` | Maximum concurrent connections |\n| `NEURALDB_SHARED_BUFFERS` | `256MB` | Row store page cache |\n| `NEURALDB_VECTOR_BUFFER` | `512MB` | Vector index memory |\n| `NEURALDB_WAL_LEVEL` | `replica` | WAL level (`minimal`, `replica`, `logical`) |\n\n## Available Tags\n\n| Tag | Description |\n|-----|-------------|\n| `latest` | Latest stable release |\n| `1.0` | Specific major version |\n| `1.0.3` | Specific patch version |\n| `nightly` | Nightly build from main branch |\n| `1.0-alpine` | Alpine-based image (smaller, less glibc compat) |\n\n## Volumes\n\nNeuralDB stores data in `/var/lib/neuraldb/data` inside the container. Always mount a named volume or bind mount to persist data:\n\n```bash\n# Named volume (recommended)\ndocker volume create neuraldb_data\ndocker run -v neuraldb_data:/var/lib/neuraldb/data neuraldb/neuraldb:latest\n\n# Bind mount\ndocker run -v /srv/neuraldb:/var/lib/neuraldb/data neuraldb/neuraldb:latest\n```\n\nThe data directory includes:\n- `base/` \u2014 table and index data\n- `vectors/` \u2014 HNSW graph files and raw vector data\n- `wal/` \u2014 write-ahead log segments\n- `neuraldb.conf` \u2014 runtime configuration (editable)\n\n## docker-compose Setup\n\nA production-grade docker-compose file for NeuralDB with automatic backup:\n\n```yaml\n# docker-compose.yml\nversion: '3.9'\n\nservices:\n neuraldb:\n image: neuraldb/neuraldb:1.0\n container_name: neuraldb\n restart: unless-stopped\n ports:\n - \"127.0.0.1:5432:5432\" # bind to localhost only \u2014 use nginx for external access\n environment:\n NEURALDB_PASSWORD: ${NEURALDB_PASSWORD}\n NEURALDB_USER: ${NEURALDB_USER:-neuraldb}\n NEURALDB_DB: ${NEURALDB_DB:-neuraldb}\n NEURALDB_SHARED_BUFFERS: \"4GB\"\n NEURALDB_VECTOR_BUFFER: \"8GB\"\n NEURALDB_MAX_CONNECTIONS: \"200\"\n volumes:\n - neuraldb_data:/var/lib/neuraldb/data\n - neuraldb_wal:/var/lib/neuraldb/wal\n - ./neuraldb.conf:/etc/neuraldb/neuraldb.conf:ro # optional custom config\n shm_size: '2gb' # increase shared memory for large sort operations\n healthcheck:\n test: [\"CMD-SHELL\", \"pg_isready -U ${NEURALDB_USER:-neuraldb}\"]\n interval: 10s\n timeout: 5s\n retries: 5\n deploy:\n resources:\n limits:\n memory: 16G\n reservations:\n memory: 8G\n\n neuraldb-backup:\n image: neuraldb/neuraldb-backup:latest\n environment:\n NEURALDB_HOST: neuraldb\n NEURALDB_PASSWORD: ${NEURALDB_PASSWORD}\n S3_BUCKET: ${BACKUP_S3_BUCKET}\n S3_PREFIX: backups/neuraldb/\n SCHEDULE: \"0 2 * * *\" # 2am daily\n depends_on:\n neuraldb:\n condition: service_healthy\n\nvolumes:\n neuraldb_data:\n driver: local\n driver_opts:\n type: none\n o: bind\n device: /srv/neuraldb/data\n neuraldb_wal:\n driver: local\n driver_opts:\n type: none\n o: bind\n device: /srv/neuraldb/wal\n```\n\nStart with:\n\n```bash\necho \"NEURALDB_PASSWORD=$(openssl rand -base64 32)\" > .env\ndocker-compose up -d\n```\n\n## Initialisation Scripts\n\nPlace `.sql` or `.sh` scripts in `/docker-entrypoint-initdb.d/` to run them on first startup:\n\n```bash\ndocker run \\\n -v ./init-scripts:/docker-entrypoint-initdb.d:ro \\\n -e NEURALDB_PASSWORD=mypassword \\\n neuraldb/neuraldb:latest\n```\n\n```sql\n-- init-scripts/01-schema.sql\nCREATE TABLE documents (\n id UUID PRIMARY KEY DEFAULT gen_random_uuid(),\n content TEXT NOT NULL,\n embedding VECTOR(1536),\n created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()\n);\n\nCREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops);\n```\n\n## Memory Tuning\n\nFor production, size the container memory based on your dataset:\n\n```\nRecommended memory = shared_buffers + vector_buffer + (max_connections \u00d7 work_mem) + OS overhead\n```\n\nFor a typical RAG application (5M documents, 1536 dimensions):\n- `vector_buffer` \u2248 5M \u00d7 1536 \u00d7 4B \u00d7 1.3 = ~40 GB\n- `shared_buffers` = 8 GB\n- `work_mem` \u00d7 connections = 128MB \u00d7 50 = 6.4 GB\n- **Total**: ~56 GB \u2014 provision a 64 GB container\n\n## Upgrading\n\n```bash\n# Pull the new image\ndocker pull neuraldb/neuraldb:1.1\n\n# Stop the current container (data is safe in the volume)\ndocker stop neuraldb && docker rm neuraldb"
|
|
},
|
|
{
|
|
"file": "pages/install-kubernetes.md",
|
|
"title": "Kubernetes",
|
|
"section-id": "installation",
|
|
"keywords": "Kubernetes, Helm, StatefulSet, PVC, k8s, cluster, deployment",
|
|
"description": "Deploying NeuralDB on Kubernetes using the official Helm chart and StatefulSets",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Kubernetes\n\nThe recommended way to run NeuralDB on Kubernetes is via the official Helm chart. The chart deploys NeuralDB as a StatefulSet with persistent volume claims, and supports both standalone and high-availability configurations.\n\n## Prerequisites\n\n- Kubernetes 1.27+\n- Helm 3.x\n- A storage class that supports `ReadWriteOnce` PVCs (most cloud providers support this)\n- At least 4 CPU cores and 8 GB RAM per NeuralDB node\n\n## Installing the Helm Chart\n\n```bash\n# Add the NeuralDB Helm repository\nhelm repo add neuraldb https://charts.neuraldb.io\nhelm repo update\n\n# Create a namespace\nkubectl create namespace neuraldb\n\n# Install the chart\nhelm install neuraldb neuraldb/neuraldb \\\n --namespace neuraldb \\\n --set auth.password=mysecretpassword \\\n --set persistence.size=100Gi\n```\n\n## Chart Configuration\n\nCreate a `values.yaml` file for production settings:\n\n```yaml\n# values.yaml\n\nimage:\n repository: neuraldb/neuraldb\n tag: \"1.0\"\n pullPolicy: IfNotPresent\n\nauth:\n # Set via --set auth.password=... or a pre-existing secret\n existingSecret: \"\"\n secretKey: \"neuraldb-password\"\n\nreplicaCount: 1 # primary nodes (use 1 for standalone)\nreadReplicaCount: 2 # read replicas\n\nresources:\n requests:\n cpu: \"2\"\n memory: \"8Gi\"\n limits:\n cpu: \"8\"\n memory: \"32Gi\"\n\npersistence:\n enabled: true\n storageClass: \"fast-ssd\" # use a fast SSD storage class\n size: 500Gi\n walSize: 50Gi # separate PVC for WAL\n\nvectorBuffer: \"16Gi\" # memory for HNSW index\nsharedBuffers: \"8Gi\" # row store page cache\nmaxConnections: 200\n\nservice:\n type: ClusterIP\n port: 5432\n\n# High-availability configuration\nha:\n enabled: true\n replication:\n mode: synchronous # 'synchronous' or 'asynchronous'\n synchronousCommit: \"on\"\n\nbackup:\n enabled: true\n schedule: \"0 2 * * *\"\n s3:\n bucket: my-neuraldb-backups\n region: us-east-1\n existingSecret: aws-credentials\n\nmonitoring:\n enabled: true\n serviceMonitor:\n enabled: true # requires Prometheus Operator\n```\n\nApply the values:\n\n```bash\nhelm install neuraldb neuraldb/neuraldb \\\n --namespace neuraldb \\\n -f values.yaml \\\n --set auth.password=$(openssl rand -base64 32)\n```\n\n## StatefulSet Details\n\nThe chart deploys a `StatefulSet` with:\n\n- One pod per replica (primary + read replicas)\n- Two PVCs per pod: data volume and WAL volume\n- An init container that configures replication on startup\n\n```yaml\n# Example pod spec (simplified)\nspec:\n containers:\n - name: neuraldb\n image: neuraldb/neuraldb:1.0\n ports:\n - containerPort: 5432\n resources:\n requests:\n memory: \"8Gi\"\n cpu: \"2\"\n volumeMounts:\n - name: data\n mountPath: /var/lib/neuraldb/data\n - name: wal\n mountPath: /var/lib/neuraldb/wal\n livenessProbe:\n exec:\n command: [\"pg_isready\", \"-U\", \"neuraldb\"]\n initialDelaySeconds: 30\n periodSeconds: 10\n readinessProbe:\n exec:\n command: [\"pg_isready\", \"-U\", \"neuraldb\"]\n initialDelaySeconds: 5\n periodSeconds: 5\n```\n\n## Services\n\nThe chart creates three Kubernetes services:\n\n| Service | Type | Port | Description |\n|---------|------|------|-------------|\n| `neuraldb-primary` | ClusterIP | 5432 | Primary \u2014 reads + writes |\n| `neuraldb-replica` | ClusterIP | 5432 | Read replicas \u2014 reads only |\n| `neuraldb-headless` | Headless | 5432 | For StatefulSet pod discovery |\n\nConnect to the primary:\n\n```bash\nkubectl port-forward svc/neuraldb-primary 5432:5432 -n neuraldb\npsql -h localhost -U neuraldb\n```\n\n## Persistent Volume Claims\n\nEach pod gets two PVCs:\n\n```yaml\nvolumeClaimTemplates:\n- metadata:\n name: data\n spec:\n accessModes: [\"ReadWriteOnce\"]\n storageClassName: fast-ssd\n resources:\n requests:\n storage: 500Gi\n- metadata:\n name: wal\n spec:\n accessModes: [\"ReadWriteOnce\"]\n storageClassName: fast-ssd\n resources:\n requests:\n storage: 50Gi\n```\n\nUse a **fast-ssd** storage class (AWS `gp3`, GCP `pd-ssd`, Azure `Premium_LRS`) for the data and WAL volumes. Spinning disks are not supported in production.\n\n## Secrets Management\n\nStore the NeuralDB password in a Kubernetes secret:\n\n```bash\nkubectl create secret generic neuraldb-credentials \\\n --namespace neuraldb \\\n --from-literal=password=$(openssl rand -base64 32)\n```\n\nReference it in `values.yaml`:\n\n```yaml\nauth:\n existingSecret: neuraldb-credentials\n secretKey: password\n```\n\nFor larger installations, use an external secrets manager (HashiCorp Vault, AWS Secrets Manager) with the External Secrets Operator.\n\n## Scaling Read Replicas\n\nScale the number of read replicas without downtime:\n\n```bash\nhelm upgrade neuraldb neuraldb/neuraldb \\\n --namespace neuraldb \\\n --set readReplicaCount=4\n```\n\nThe new replica pods will join the replication stream automatically.\n\n## Upgrading\n\n```bash\nhelm repo update\nhelm upgrade neuraldb neuraldb/neuraldb \\\n --namespace neuraldb \\\n -f values.yaml \\\n --set auth.existingSecret=neuraldb-credentials\n```\n\nT"
|
|
},
|
|
{
|
|
"file": "pages/install-local.md",
|
|
"title": "Local Development",
|
|
"section-id": "installation",
|
|
"keywords": "local, development, binary, homebrew, winget, install, macOS, Linux, Windows",
|
|
"description": "Installing NeuralDB locally for development using binaries, Homebrew, or winget",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Local Development\n\nFor local development, you can run NeuralDB as a native binary without Docker. This provides lower latency for development workflows and avoids container overhead.\n\n## System Requirements\n\n| Platform | Minimum | Recommended |\n|----------|---------|-------------|\n| macOS | 13.0 (Ventura) | 14.x+ |\n| Linux | Ubuntu 22.04 / RHEL 9 | Ubuntu 24.04 |\n| Windows | Windows 10 22H2 | Windows 11 |\n| CPU | x86-64 or ARM64 | ARM64 (Apple Silicon) |\n| RAM | 4 GB | 16 GB+ |\n| Disk | 2 GB free | SSD recommended |\n\n## macOS\n\n### Homebrew (Recommended)\n\n```bash\nbrew tap neuraldb/tap\nbrew install neuraldb\n\n# Start as a service (auto-restart on login)\nbrew services start neuraldb\n\n# Or start manually (foreground)\nneuraldb start\n\n# Check status\nneuraldb status\n```\n\nThe Homebrew formula installs:\n- `neuraldb` \u2014 the server binary\n- `neuraldb-cli` \u2014 an enhanced SQL shell (psql-compatible)\n- Configuration at `$(brew --prefix)/etc/neuraldb/neuraldb.conf`\n- Data directory at `$(brew --prefix)/var/neuraldb`\n\n### Direct Binary\n\n```bash\n# Apple Silicon (M1/M2/M3/M4)\ncurl -LO https://releases.neuraldb.io/1.0/neuraldb-macos-arm64.tar.gz\ntar -xzf neuraldb-macos-arm64.tar.gz\nsudo mv neuraldb /usr/local/bin/\nsudo mv neuraldb-cli /usr/local/bin/\n\n# Intel Mac\ncurl -LO https://releases.neuraldb.io/1.0/neuraldb-macos-amd64.tar.gz\ntar -xzf neuraldb-macos-amd64.tar.gz\nsudo mv neuraldb /usr/local/bin/\nsudo mv neuraldb-cli /usr/local/bin/\n```\n\n## Linux\n\n### Ubuntu / Debian\n\n```bash\n# Add the NeuralDB APT repository\ncurl -fsSL https://packages.neuraldb.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/neuraldb-keyring.gpg\necho \"deb [signed-by=/usr/share/keyrings/neuraldb-keyring.gpg] https://packages.neuraldb.io/apt stable main\" \\\n | sudo tee /etc/apt/sources.list.d/neuraldb.list\n\nsudo apt update\nsudo apt install -y neuraldb\n\n# Start and enable the service\nsudo systemctl enable --now neuraldb\n```\n\n### RHEL / Fedora / CentOS\n\n```bash\n# Add the NeuralDB YUM repository\nsudo rpm --import https://packages.neuraldb.io/gpg\nsudo tee /etc/yum.repos.d/neuraldb.repo <<'EOF'\n[neuraldb]\nname=NeuralDB Repository\nbaseurl=https://packages.neuraldb.io/rpm/stable\nenabled=1\ngpgcheck=1\ngpgkey=https://packages.neuraldb.io/gpg\nEOF\n\nsudo dnf install -y neuraldb\nsudo systemctl enable --now neuraldb\n```\n\n### Direct Binary (Linux)\n\n```bash\n# x86-64\ncurl -LO https://releases.neuraldb.io/1.0/neuraldb-linux-amd64.tar.gz\ntar -xzf neuraldb-linux-amd64.tar.gz\nsudo mv neuraldb neuraldb-cli /usr/local/bin/\n```\n\n## Windows\n\n### winget\n\n```powershell\nwinget install NeuralDB.NeuralDB\n```\n\nThis installs NeuralDB and registers it as a Windows Service. It starts automatically after installation.\n\n### Chocolatey\n\n```powershell\nchoco install neuraldb\n```\n\n### MSI Installer\n\nDownload the MSI installer from [neuraldb.io/download](https://neuraldb.io/download) and run it. The installer:\n1. Installs `neuraldb.exe` and `neuraldb-cli.exe` to `C:\\Program Files\\NeuralDB\\`\n2. Adds them to `PATH`\n3. Creates a `NeuralDB` Windows Service\n4. Initialises the data directory at `%APPDATA%\\NeuralDB\\data`\n\n## First-Time Setup\n\nAfter installation, initialise the database:\n\n```bash\n# Linux / macOS\nneuraldb init\nneuraldb start\n\n# Connect\nneuraldb-cli\n# psql prompt: neuraldb=#\n```\n\n### Change the Default Password\n\n```sql\nALTER USER neuraldb PASSWORD 'your-new-password';\n```\n\n### Create a Development Database\n\n```sql\nCREATE DATABASE myapp;\n\\c myapp\n\nCREATE TABLE documents (\n id UUID PRIMARY KEY DEFAULT gen_random_uuid(),\n content TEXT,\n embedding VECTOR(1536)\n);\n\nCREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops);\n```\n\n## Configuration\n\nThe default configuration file locations:\n\n| Platform | Path |\n|----------|------|\n| macOS (Homebrew) | `$(brew --prefix)/etc/neuraldb/neuraldb.conf` |\n| Linux | `/etc/neuraldb/neuraldb.conf` |\n| Windows | `%PROGRAMDATA%\\NeuralDB\\neuraldb.conf` |\n\nFor local development, create a `neuraldb.conf` in your project directory and point to it:\n\n```bash\nneuraldb start --config ./neuraldb.conf\n```\n\nUseful development settings:\n\n```ini\n# neuraldb.conf (development)\nlisten_addresses = 'localhost'\nport = 5432\nmax_connections = 50\nshared_buffers = 256MB\nvector_buffer = 1GB\nlog_min_duration_statement = 100 # log slow queries (>100ms)\nlog_statement = 'all' # log all SQL (useful for debugging)\n```\n\n## Uninstalling\n\n```bash\n# macOS\nbrew services stop neuraldb\nbrew uninstall neuraldb\n\n# Ubuntu\nsudo systemctl stop neuraldb\nsudo apt remove neuraldb\n\n# Windows\nwinget uninstall NeuralDB.NeuralDB\n# Or: Settings \u2192 Apps \u2192 Installed Apps \u2192 NeuralDB \u2192 Uninstall\n```"
|
|
},
|
|
{
|
|
"file": "pages/nql-aggregations.md",
|
|
"title": "Aggregations",
|
|
"section-id": "query-language",
|
|
"keywords": "aggregations, GROUP BY, COUNT, SUM, vectors, AVG, centroid, analytics",
|
|
"description": "Aggregating data in NQL including GROUP BY, COUNT, SUM, and vector-specific aggregation functions",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Aggregations\n\nNQL supports the full SQL aggregation toolkit, extended with vector-specific aggregate functions for centroid computation, clustering, and semantic analytics.\n\n## Standard Aggregations\n\nAll standard SQL aggregate functions work as expected:\n\n```sql\n-- Count documents by category\nSELECT category, COUNT(*) AS doc_count\nFROM documents\nGROUP BY category\nORDER BY doc_count DESC;\n\n-- Average price by category\nSELECT category,\n COUNT(*) AS products,\n AVG(price) AS avg_price,\n MIN(price) AS min_price,\n MAX(price) AS max_price,\n SUM(stock * price) AS inventory_value\nFROM products\nWHERE available = true\nGROUP BY category\nORDER BY inventory_value DESC;\n```\n\n## Vector Aggregations\n\n### `AVG(embedding)` \u2014 Centroid Computation\n\nCompute the centroid (average vector) of a group:\n\n```sql\n-- Centroid of all \"technology\" documents\nSELECT AVG(embedding) AS centroid\nFROM documents\nWHERE category = 'technology';\n```\n\nUse centroids to find documents representative of a cluster:\n\n```sql\nWITH centroid AS (\n SELECT AVG(embedding) AS c FROM documents WHERE category = 'technology'\n)\nSELECT id, title, 1 - (embedding <=> centroid.c) AS similarity_to_centroid\nFROM documents, centroid\nWHERE category = 'technology'\nORDER BY embedding <=> centroid.c\nLIMIT 10;\n```\n\n### `vector_centroid(embedding)` \u2014 Weighted Centroid\n\nCompute a weighted centroid using a score column:\n\n```sql\n-- Weighted centroid by rating (higher-rated items pull more)\nSELECT vector_centroid(embedding, rating) AS weighted_centroid\nFROM products\nWHERE category = 'electronics';\n```\n\n### `vector_agg_concat(embedding)` \u2014 Vector Array\n\nCollect vectors into an array for downstream processing:\n\n```sql\nSELECT category, vector_agg_concat(embedding) AS all_embeddings\nFROM documents\nGROUP BY category;\n```\n\n## GROUP BY with Vector Search\n\nFind the best document in each category for a given query:\n\n```sql\nSELECT DISTINCT ON (category)\n id, category, title, 1 - (embedding <=> $1) AS similarity\nFROM documents\nWHERE embedding IS NOT NULL\nORDER BY category, embedding <=> $1;\n```\n\nOr using a lateral join for more control:\n\n```sql\nSELECT cat.category, top_doc.id, top_doc.title, top_doc.similarity\nFROM (SELECT DISTINCT category FROM documents) cat,\nLATERAL (\n SELECT id, title, 1 - (embedding <=> $1) AS similarity\n FROM documents\n WHERE category = cat.category\n ORDER BY embedding <=> $1\n LIMIT 1\n) top_doc;\n```\n\n## Window Functions\n\nUse window functions to rank results within partitions:\n\n```sql\n-- Rank documents by similarity within each category\nSELECT\n id, title, category,\n 1 - (embedding <=> $1) AS similarity,\n RANK() OVER (\n PARTITION BY category\n ORDER BY embedding <=> $1\n ) AS rank_in_category\nFROM documents\nWHERE 1 - (embedding <=> $1) > 0.5\nORDER BY category, rank_in_category;\n```\n\nRolling average similarity over time:\n\n```sql\nSELECT\n date_trunc('day', created_at) AS day,\n AVG(1 - (embedding <=> $1)) AS avg_daily_similarity,\n AVG(AVG(1 - (embedding <=> $1))) OVER (\n ORDER BY date_trunc('day', created_at)\n ROWS BETWEEN 6 PRECEDING AND CURRENT ROW\n ) AS rolling_7d_avg\nFROM documents\nGROUP BY day\nORDER BY day;\n```\n\n## Clustering with GROUP BY\n\nPerform k-means style clustering by assigning documents to their nearest centroid:\n\n```sql\n-- Given pre-computed centroids in a centroids table:\nSELECT d.id, d.content,\n c.cluster_id,\n (d.embedding <=> c.centroid) AS distance_to_centroid\nFROM documents d\nCROSS JOIN LATERAL (\n SELECT cluster_id, centroid\n FROM centroids\n ORDER BY d.embedding <=> centroid\n LIMIT 1\n) c;\n```\n\n## HAVING with Vector Conditions\n\n```sql\n-- Categories where the average intra-category similarity is high (tight clusters)\nSELECT category,\n COUNT(*) AS doc_count,\n 1 - AVG(embedding <=> (SELECT AVG(e2.embedding) FROM documents e2 WHERE e2.category = e.category)) AS cohesion\nFROM documents e\nGROUP BY category\nHAVING COUNT(*) > 10\nORDER BY cohesion DESC;\n```\n\n## Time-Series Analytics\n\nAnalyse how semantic content shifts over time:\n\n```sql\n-- Daily semantic drift: how different is today's content from last week's?\nWITH weekly_centroids AS (\n SELECT\n date_trunc('week', created_at) AS week,\n AVG(embedding) AS centroid\n FROM documents\n GROUP BY week\n)\nSELECT\n w1.week,\n 1 - (w1.centroid <=> w2.centroid) AS similarity_to_prev_week\nFROM weekly_centroids w1\nLEFT JOIN weekly_centroids w2\n ON w2.week = w1.week - INTERVAL '1 week'\nORDER BY w1.week;\n```\n\n## JSON Aggregation with Vectors\n\nCombine JSON aggregation with vector results:\n\n```sql\nSELECT\n category,\n COUNT(*) AS total,\n AVG(price) AS avg_price,\n JSON_AGG(\n JSON_BUILD_OBJECT('id', id, 'name', name, 'similarity', 1 - (embedding <=> $1))\n ORDER BY embedding <=> $1\n ) FILTER (WHERE ROW_NUMBER() OVER (PARTITION BY category ORDER BY embedding <=> $1) <= 3)\n AS top_3_per_category\nFROM products\nWHERE available = true\nGROUP BY category;\n```\n\n## ROLLUP and CUBE\n\nStandard SQL ROLLUP and CUBE work for hierarchical aggregati"
|
|
},
|
|
{
|
|
"file": "pages/nql-basics.md",
|
|
"title": "NQL Basics",
|
|
"section-id": "query-language",
|
|
"keywords": "NQL, NeuralDB Query Language, SQL, syntax, basics, queries",
|
|
"description": "Introduction to NeuralDB Query Language (NQL) \u2014 syntax, data types, and basic operations",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# NQL Basics\n\nNQL (NeuralDB Query Language) is a superset of standard SQL. Every valid SQL statement is also valid NQL. NQL adds extensions for vector operations, embedding generation, and semantic search primitives.\n\nIf you know SQL, you already know most of NQL. This page covers the NQL-specific additions and the data types introduced for AI workloads.\n\n## Connecting\n\nNeuralDB speaks the PostgreSQL wire protocol. Connect with any PostgreSQL client:\n\n```bash\n# psql\npsql -h localhost -p 5432 -U neuraldb -d mydb\n\n# neuraldb-cli (enhanced interactive shell)\nneuraldb-cli -h localhost\n```\n\nConnection string format:\n\n```\npostgresql://[user[:password]@][host][:port][/dbname][?param=value...]\n```\n\n## Data Types\n\n### VECTOR(n)\n\nThe core NQL extension. Stores a fixed-length array of 32-bit floats representing a vector embedding:\n\n```sql\n-- Declare a vector column with 1536 dimensions (OpenAI ada-002 output)\nCREATE TABLE documents (\n id UUID PRIMARY KEY DEFAULT gen_random_uuid(),\n content TEXT NOT NULL,\n embedding VECTOR(1536)\n);\n```\n\nInsert a vector by providing a bracketed float array:\n\n```sql\nINSERT INTO documents (content, embedding)\nVALUES ('Hello, world', '[0.1, 0.2, 0.3, ...]'); -- 1536 values\n```\n\n### HALFVEC(n)\n\nA 16-bit float variant of VECTOR. Half the memory, slight precision loss. Useful when vector_buffer is a constraint:\n\n```sql\nembedding HALFVEC(1536)\n```\n\n### SPARSEVEC(n)\n\nSparse vector representation \u2014 stores only non-zero elements. Efficient for high-dimensional but sparse vectors (e.g., BM25 term-frequency vectors):\n\n```sql\nbm25_vector SPARSEVEC(30000)\n```\n\n## Basic CRUD\n\n### Creating Tables\n\n```sql\nCREATE TABLE products (\n id UUID PRIMARY KEY DEFAULT gen_random_uuid(),\n name TEXT NOT NULL,\n description TEXT,\n category TEXT,\n price DECIMAL(10, 2),\n stock INTEGER DEFAULT 0,\n embedding VECTOR(1536),\n created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()\n);\n```\n\n### Inserting Data\n\n```sql\nINSERT INTO products (name, description, category, price, stock, embedding)\nVALUES (\n 'Wireless Headphones',\n 'Premium noise-cancelling wireless headphones with 30-hour battery',\n 'electronics',\n 299.99,\n 150,\n '[0.023, -0.187, 0.412, ...]' -- 1536 floats from your embedding model\n);\n```\n\n### Reading Data\n\nStandard SQL SELECT works as expected:\n\n```sql\nSELECT id, name, price FROM products WHERE category = 'electronics';\nSELECT * FROM products WHERE price BETWEEN 50 AND 300 AND stock > 0;\n```\n\n### Updating Data\n\n```sql\nUPDATE products SET price = 279.99, embedding = '[...]' WHERE id = $1;\n```\n\nWhen updating the `embedding` column, the HNSW index is updated atomically.\n\n### Deleting Data\n\n```sql\nDELETE FROM products WHERE id = $1;\n```\n\n## Creating Vector Indexes\n\nWithout an index, vector similarity queries perform exact linear scans (O(n)). Create an HNSW index for sub-linear performance:\n\n```sql\n-- Cosine similarity (most common for text embeddings)\nCREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops);\n\n-- Euclidean distance\nCREATE INDEX ON documents USING hnsw (embedding vector_l2_ops);\n\n-- Dot product (for recommendation systems)\nCREATE INDEX ON documents USING hnsw (embedding vector_ip_ops);\n```\n\nBuild an index on an existing large table in parallel:\n\n```sql\nSET max_parallel_maintenance_workers = 8;\nCREATE INDEX CONCURRENTLY ON documents USING hnsw (embedding vector_cosine_ops);\n```\n\n## Basic Vector Queries\n\n### Find Similar Documents\n\n```sql\nSELECT id, content, 1 - (embedding <=> $1) AS similarity\nFROM documents\nORDER BY embedding <=> $1\nLIMIT 10;\n```\n\nThe `<=>` operator is cosine distance (lower = more similar). Subtract from 1 for a similarity score (higher = more similar).\n\n### Distance Operators\n\n| Operator | Distance metric | Index ops |\n|----------|----------------|-----------|\n| `<=>` | Cosine distance | `vector_cosine_ops` |\n| `<->` | Euclidean (L2) distance | `vector_l2_ops` |\n| `<#>` | Negative dot product | `vector_ip_ops` |\n\n### Distance Threshold\n\n```sql\n-- Only return results with cosine similarity > 0.8\nSELECT id, content, 1 - (embedding <=> $1) AS similarity\nFROM documents\nWHERE 1 - (embedding <=> $1) > 0.8\nORDER BY embedding <=> $1\nLIMIT 20;\n```\n\n**Note:** The `WHERE 1 - (embedding <=> $1) > 0.8` condition is evaluated after the ANN search, not before. Use `LIMIT` generously enough to capture all relevant results before the threshold filter.\n\n## NQL Functions\n\n### `to_vector(text)`\n\nConvert a string literal to a VECTOR:\n\n```sql\nSELECT to_vector('[0.1, 0.2, 0.3]')::VECTOR(3);\n```\n\n### `vector_dims(v)`\n\nReturn the number of dimensions:\n\n```sql\nSELECT vector_dims(embedding) FROM documents LIMIT 1;\n-- Returns: 1536\n```\n\n### `vector_norm(v)`\n\nReturn the L2 norm of a vector:\n\n```sql\nSELECT vector_norm(embedding) FROM documents LIMIT 5;\n```\n\n### `cosine_similarity(a, b)`, `l2_distance(a, b)`, `dot_product(a, b)`\n\nNamed function alternatives to the operators:\n\n```sql\nSELECT cosine_similarity(embedding, $1) AS similarity\nFROM documents\nORDER BY similarity DESC\nLIMIT 10;\n```\n\n#"
|
|
},
|
|
{
|
|
"file": "pages/nql-hybrid.md",
|
|
"title": "Hybrid Queries",
|
|
"section-id": "query-language",
|
|
"keywords": "hybrid queries, vector, relational, filters, combined, semantic search, metadata",
|
|
"description": "Combining vector similarity and relational filters in NQL hybrid queries",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Hybrid Queries\n\nHybrid queries combine vector similarity search with relational filter predicates in a single SQL statement. The NeuralDB query planner handles the execution strategy \u2014 you write normal SQL with vector operators.\n\n## Basic Hybrid Query\n\nFind the 10 most semantically similar products that are in the \"electronics\" category and in stock:\n\n```sql\nSELECT id, name, price, 1 - (embedding <=> $1) AS similarity\nFROM products\nWHERE category = 'electronics'\n AND stock > 0\n AND price < 500\nORDER BY embedding <=> $1\nLIMIT 10;\n```\n\nNeuralDB automatically determines whether to:\n1. **Pre-filter**: apply relational conditions first, then search the filtered set\n2. **Post-filter**: run ANN search, then apply conditions to the top-k results\n\nThe decision is based on the selectivity of the relational predicates.\n\n## Query Planner Hints\n\nOverride the planner's strategy:\n\n```sql\n-- Force pre-filter (good when relational filter is very selective)\nSELECT /*+ PREFILTER */ id, name, score\nFROM (\n SELECT id, name, 1 - (embedding <=> $1) AS score\n FROM products\n WHERE category = 'electronics' -- very selective: 2% of rows\n) sub\nORDER BY score DESC\nLIMIT 10;\n\n-- Force post-filter (good when relational filter is weakly selective)\nSELECT /*+ POSTFILTER */ id, name, 1 - (embedding <=> $1) AS score\nFROM products\nWHERE price < 500 -- weakly selective: 80% of rows\nORDER BY embedding <=> $1\nLIMIT 10;\n```\n\n## Filtering by Multiple Conditions\n\n```sql\nSELECT id, name, description,\n 1 - (embedding <=> $1) AS similarity,\n price,\n rating\nFROM products\nWHERE category = ANY($2) -- multi-category filter\n AND price BETWEEN $3 AND $4\n AND rating >= 4.0\n AND discontinued = false\n AND created_at > NOW() - INTERVAL '1 year'\nORDER BY embedding <=> $1\nLIMIT 20;\n-- $1 = query embedding\n-- $2 = ['electronics', 'computers']\n-- $3 = 50, $4 = 1000\n```\n\n## Hybrid Full-Text + Vector (BM25)\n\nCombine traditional full-text search with vector similarity using Reciprocal Rank Fusion (RRF):\n\n```sql\nWITH vector_search AS (\n SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $1) AS rank\n FROM documents\n ORDER BY embedding <=> $1\n LIMIT 100\n),\nfts_search AS (\n SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank_cd(tsv, query) DESC) AS rank\n FROM documents, to_tsquery('english', $2) query\n WHERE tsv @@ query\n ORDER BY ts_rank_cd(tsv, query) DESC\n LIMIT 100\n),\nrrf AS (\n SELECT\n COALESCE(v.id, f.id) AS id,\n (COALESCE(1.0 / (60 + v.rank), 0) + COALESCE(1.0 / (60 + f.rank), 0)) AS rrf_score\n FROM vector_search v\n FULL OUTER JOIN fts_search f ON v.id = f.id\n)\nSELECT d.id, d.content, rrf.rrf_score\nFROM rrf\nJOIN documents d ON d.id = rrf.id\nORDER BY rrf_score DESC\nLIMIT 10;\n```\n\nNQL also provides a built-in `HYBRID_SEARCH` function:\n\n```sql\nSELECT id, content, score\nFROM HYBRID_SEARCH(\n table := 'documents',\n vector_column := 'embedding',\n tsv_column := 'tsv',\n query_vector := $1,\n query_text := $2,\n top_k := 10,\n rrf_k := 60,\n vector_weight := 0.6,\n text_weight := 0.4\n);\n```\n\n## Joining Vector Results with Other Tables\n\n```sql\nSELECT\n p.id,\n p.name,\n p.price,\n c.name AS category_name,\n u.display_name AS seller,\n 1 - (p.embedding <=> $1) AS similarity\nFROM products p\nJOIN categories c ON c.id = p.category_id\nJOIN users u ON u.id = p.seller_id\nWHERE p.available = true\n AND c.slug = ANY($2)\n AND u.verified = true\nORDER BY p.embedding <=> $1\nLIMIT 15;\n```\n\n## Subquery Vectors\n\nUse a subquery to dynamically compute a query vector from existing data:\n\n```sql\n-- Find products similar to product #123\nSELECT id, name, 1 - (embedding <=> ref.embedding) AS similarity\nFROM products,\n (SELECT embedding FROM products WHERE id = $1) ref\nWHERE id != $1\nORDER BY embedding <=> ref.embedding\nLIMIT 10;\n```\n\n## Tenant-Scoped Search\n\nFor multi-tenant applications, always include tenant filters:\n\n```sql\nSELECT id, content, 1 - (embedding <=> $1) AS similarity\nFROM documents\nWHERE tenant_id = $2 -- partition pruning if SHARD BY tenant_id\n AND embedding IS NOT NULL\nORDER BY embedding <=> $1\nLIMIT 10;\n```\n\nIf the table is sharded by `tenant_id`, this query runs entirely on the correct shard without cross-shard coordination.\n\n## Composite Scoring\n\nCombine vector similarity with relational signals:\n\n```sql\nSELECT\n id,\n name,\n price,\n rating,\n -- Weighted composite score: 70% semantic, 20% rating, 10% recency\n (0.7 * (1 - (embedding <=> $1))\n + 0.2 * (rating / 5.0)\n + 0.1 * (1 - EXTRACT(DAYS FROM NOW() - created_at) / 365.0)\n ) AS composite_score\nFROM products\nWHERE available = true\n AND price < $2\nORDER BY composite_score DESC\nLIMIT 20;\n```\n\n## Pagination\n\nCursor-based pagination for vector results:\n\n```sql\n-- Page 1\nSELECT id, name, (embedding <=> $1) AS dist\nFROM products\nWHERE available = true\nORDER BY dist, id -- secondary sort by id for stable pagination\nLIMIT 20;\n\n-- Page 2 (cursor: last dist and id from page 1)\nSELECT id, name, (embedding <=> $1) AS dist\nFROM products\nWHERE available "
|
|
},
|
|
{
|
|
"file": "pages/nql-transactions.md",
|
|
"title": "Transactions",
|
|
"section-id": "query-language",
|
|
"keywords": "transactions, ACID, isolation levels, MVCC, BEGIN, COMMIT, ROLLBACK",
|
|
"description": "ACID transactions in NeuralDB \u2014 isolation levels, MVCC, savepoints, and advisory locks",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Transactions\n\nNeuralDB provides full ACID transactions with MVCC (Multi-Version Concurrency Control). Unlike most vector databases, NeuralDB guarantees atomicity across both relational and vector data within a single transaction.\n\n## ACID Guarantees\n\n| Property | Guarantee |\n|----------|---------|\n| **Atomicity** | All operations in a transaction succeed or all are rolled back \u2014 including vector index updates |\n| **Consistency** | Constraints (foreign keys, unique indexes, not null) are enforced at commit time |\n| **Isolation** | Concurrent transactions do not see each other's uncommitted changes |\n| **Durability** | Committed transactions survive crashes via the WAL |\n\n## Basic Transaction Syntax\n\n```sql\nBEGIN;\n\n-- Your operations here\nINSERT INTO documents (content, embedding) VALUES ($1, $2);\nUPDATE document_stats SET total_count = total_count + 1;\nINSERT INTO audit_log (action, data) VALUES ('insert', $3);\n\nCOMMIT;\n```\n\nOn error, roll back:\n\n```sql\nBEGIN;\n\nINSERT INTO documents (content, embedding) VALUES ($1, $2);\n\n-- Something went wrong\nROLLBACK;\n```\n\n## Isolation Levels\n\nNeuralDB supports four isolation levels. Set them with `SET TRANSACTION ISOLATION LEVEL`:\n\n```sql\nBEGIN;\nSET TRANSACTION ISOLATION LEVEL READ COMMITTED;\n-- ... your queries ...\nCOMMIT;\n```\n\n### Read Committed (Default)\n\nEach statement sees only rows committed before that statement began. Two successive reads within the same transaction may see different data if another transaction commits between them.\n\n```sql\nBEGIN;\n-- Sees all rows committed before this SELECT\nSELECT COUNT(*) FROM documents; -- Returns 1000\n\n-- Another transaction inserts and commits a row here\n\n-- Sees the new row (non-repeatable read)\nSELECT COUNT(*) FROM documents; -- Returns 1001\nCOMMIT;\n```\n\n### Repeatable Read\n\nA transaction sees only rows committed before the transaction began. Reads are stable throughout the transaction.\n\n```sql\nBEGIN ISOLATION LEVEL REPEATABLE READ;\nSELECT COUNT(*) FROM documents; -- Returns 1000\n\n-- Another transaction inserts and commits\n\nSELECT COUNT(*) FROM documents; -- Still 1000 \u2014 repeatable read\nCOMMIT;\n```\n\n### Serializable\n\nThe strictest level. Transactions execute as if they ran serially one after another. NeuralDB uses Serializable Snapshot Isolation (SSI) \u2014 it allows concurrent execution but detects and aborts transactions that would produce a non-serializable outcome.\n\n```sql\nBEGIN ISOLATION LEVEL SERIALIZABLE;\n-- ... complex read-modify-write patterns ...\nCOMMIT;\n-- May raise: ERROR: could not serialize access \u2014 retry the transaction\n```\n\n### Read Uncommitted\n\nNeuralDB maps this to Read Committed (it does not implement dirty reads).\n\n## Retry Logic\n\nSerializable transactions can fail with serialization errors. Always retry:\n\n```python\nfrom psycopg2 import errors\n\nMAX_RETRIES = 5\n\nfor attempt in range(MAX_RETRIES):\n try:\n with conn.cursor() as cur:\n cur.execute(\"BEGIN ISOLATION LEVEL SERIALIZABLE\")\n # ... your operations ...\n cur.execute(\"COMMIT\")\n break\n except errors.SerializationFailure:\n conn.rollback()\n if attempt == MAX_RETRIES - 1:\n raise\n time.sleep(0.1 * (2 ** attempt)) # exponential backoff\n```\n\n## Savepoints\n\nSavepoints allow partial rollbacks within a transaction:\n\n```sql\nBEGIN;\n\nINSERT INTO documents (content, embedding) VALUES ($1, $2);\n\nSAVEPOINT after_insert;\n\n-- Risky operation\nUPDATE document_stats SET count = count + 1 WHERE id = $3;\n\n-- Oh no, something went wrong \u2014 roll back to the savepoint\nROLLBACK TO SAVEPOINT after_insert;\n\n-- The INSERT is still pending \u2014 we can try a different approach\nUPDATE document_stats SET count = count + 1 WHERE id = $4;\n\nCOMMIT;\n```\n\n## Vector Transactions\n\nVector index updates are transactional in NeuralDB. An HNSW index entry is added atomically with the row:\n\n```sql\nBEGIN;\n\n-- Both the row and the vector index entry are inserted atomically\nINSERT INTO documents (id, content, embedding) VALUES ($1, $2, $3);\n\n-- If we ROLLBACK, neither the row nor the index entry will exist\nROLLBACK;\n\n-- After rollback, a similarity search will NOT find $1\nSELECT id FROM documents ORDER BY embedding <=> $3 LIMIT 1;\n-- $1 is not returned\n```\n\n## Long-Running Transactions\n\nAvoid long-running transactions \u2014 they:\n- Hold row-level locks, blocking other writes\n- Prevent VACUUM from reclaiming dead rows (bloat)\n- Increase the risk of deadlocks\n\nSet a statement timeout to kill runaway queries:\n\n```sql\nSET statement_timeout = '30s';\n```\n\nSet a transaction timeout:\n\n```sql\nSET idle_in_transaction_session_timeout = '5min';\n```\n\n## Deadlock Detection\n\nNeuralDB automatically detects deadlocks and aborts one of the transactions:\n\n```\nERROR: deadlock detected\nDETAIL: Process 12345 waits for ShareLock on transaction 67890; blocked by process 99999.\nHint: See server log for query details.\n```\n\nMinimise deadlock risk by always acquiring locks in the same order across all transactions.\n\n## Advisory Locks\n\nFor applicat"
|
|
},
|
|
{
|
|
"file": "pages/nql-vectors.md",
|
|
"title": "Vector Queries",
|
|
"section-id": "query-language",
|
|
"keywords": "vector queries, NEAREST, SIMILAR, cosine, dot product, euclidean, ANN",
|
|
"description": "Writing vector similarity queries in NQL \u2014 NEAREST, SIMILAR, distance operators, and recall tuning",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Vector Queries\n\nNQL extends standard SQL with operators and functions for vector similarity search. This page covers every method for querying vectors, from basic nearest-neighbour lookups to advanced recall tuning.\n\n## Distance Operators\n\nNQL provides three distance operators that double as index-acceleration hints:\n\n```sql\n-- Cosine distance (returns 0 to 2, lower = more similar)\nembedding <=> query_vector\n\n-- Euclidean (L2) distance (returns 0 to \u221e, lower = more similar)\nembedding <-> query_vector\n\n-- Negative dot product (higher inner product = more similar \u2192 negate for ORDER BY)\nembedding <#> query_vector\n```\n\nAlways pair `ORDER BY` with `LIMIT` when using distance operators \u2014 the planner uses the HNSW index only when there is an explicit `ORDER BY ... LIMIT`:\n\n```sql\n-- \u2705 Uses HNSW index\nSELECT id, content FROM documents\nORDER BY embedding <=> '[0.1, 0.2, ...]'\nLIMIT 10;\n\n-- \u274c Full scan (no ORDER BY ... LIMIT)\nSELECT id, content FROM documents\nWHERE (embedding <=> '[0.1, 0.2, ...]') < 0.3;\n```\n\n## NEAREST Clause\n\nNQL provides a syntactic alternative to `ORDER BY ... LIMIT` for nearest-neighbour queries:\n\n```sql\nSELECT id, content, score\nFROM documents\nNEAREST TO embedding = '[0.1, 0.2, ...]' USING COSINE\nTOP 10;\n```\n\nThis is equivalent to:\n\n```sql\nSELECT id, content, 1 - (embedding <=> '[0.1, 0.2, ...]') AS score\nFROM documents\nORDER BY embedding <=> '[0.1, 0.2, ...]'\nLIMIT 10;\n```\n\nThe `NEAREST TO` clause is more readable and allows NeuralDB to apply additional optimisations.\n\n### Distance Metrics in NEAREST\n\n```sql\nNEAREST TO embedding = $1 USING COSINE TOP 10\nNEAREST TO embedding = $1 USING EUCLIDEAN TOP 10\nNEAREST TO embedding = $1 USING DOT_PRODUCT TOP 10\n```\n\n## SIMILAR Clause\n\n`SIMILAR` returns results above a similarity threshold rather than a fixed count. Because the threshold is checked after the ANN search, NeuralDB must retrieve an initial candidate set. Use `LIMIT` to cap the candidates:\n\n```sql\nSELECT id, content, score\nFROM documents\nSIMILAR TO embedding = $1 USING COSINE THRESHOLD 0.75\nLIMIT 100;\n```\n\nThis returns up to 100 documents with cosine similarity \u2265 0.75.\n\n## Returning Scores\n\nInclude the distance or similarity score in results:\n\n```sql\n-- Distance (lower = more similar)\nSELECT id, content, (embedding <=> $1) AS distance\nFROM documents\nORDER BY embedding <=> $1\nLIMIT 10;\n\n-- Similarity (higher = more similar, cosine)\nSELECT id, content, 1 - (embedding <=> $1) AS similarity\nFROM documents\nORDER BY embedding <=> $1\nLIMIT 10;\n```\n\n## Querying With a Vector Literal\n\nPass vectors as SQL parameters (recommended) or literals:\n\n```sql\n-- Parameterised (prevents injection, preferred)\nSELECT id, content FROM documents\nORDER BY embedding <=> $1\nLIMIT 10;\n-- $1 = '[0.023, -0.187, 0.412, ...]'\n\n-- Inline literal (useful in SQL shells)\nSELECT id, content FROM documents\nORDER BY embedding <=> '[0.023, -0.187, 0.412]'::VECTOR(3)\nLIMIT 5;\n```\n\n## Recall Tuning\n\nThe HNSW index trades recall for performance. By default, `hnsw.ef_search = 40`, which provides ~95% recall at ~1ms latency for 10M vectors.\n\nIncrease `ef_search` for higher recall:\n\n```sql\n-- Set for the current session\nSET hnsw.ef_search = 200;\n\n-- Set for the current transaction\nBEGIN;\nSET LOCAL hnsw.ef_search = 200;\nSELECT * FROM documents ORDER BY embedding <=> $1 LIMIT 10;\nCOMMIT;\n```\n\nTypical recall vs performance trade-off (10M 1536-dim vectors, 32 vCPU):\n\n| ef_search | Recall@10 | p50 latency | QPS |\n|-----------|-----------|-------------|-----|\n| 20 | 89% | 0.7ms | 12,000 |\n| 40 | 95% | 1.2ms | 8,400 |\n| 80 | 98% | 2.1ms | 4,800 |\n| 200 | 99.5% | 4.8ms | 2,100 |\n| exact | 100% | 45ms | 220 |\n\n## Exact Search\n\nForce exact (brute-force) nearest-neighbour search, ignoring the HNSW index:\n\n```sql\nSET neuraldb.vector_scan = 'exact';\nSELECT * FROM documents ORDER BY embedding <=> $1 LIMIT 10;\nRESET neuraldb.vector_scan;\n```\n\nUse exact search when:\n- You need 100% recall (e.g., de-duplication, exact compliance checks)\n- The table has fewer than ~100k rows (exact is competitive)\n- You are benchmarking ANN recall\n\n## Bulk Vector Operations\n\n### Batch Insert\n\nUse `COPY` for high-throughput ingestion:\n\n```bash\n# Format: id\\tcontent\\tembedding\npsql -c \"\\COPY documents (id, content, embedding) FROM '/data/vectors.tsv'\"\n```\n\n### Updating Embeddings in Bulk\n\n```sql\nUPDATE documents\nSET embedding = new_embeddings.embedding\nFROM (VALUES\n ('uuid-1', '[...]'::VECTOR(1536)),\n ('uuid-2', '[...]'::VECTOR(1536))\n) AS new_embeddings(id, embedding)\nWHERE documents.id = new_embeddings.id::UUID;\n```\n\n## Multi-Vector Queries\n\nFind documents closest to ANY of multiple query vectors (OR semantics):\n\n```sql\nWITH queries AS (\n SELECT UNNEST(ARRAY['[...]'::VECTOR(1536), '[...]'::VECTOR(1536)]) AS qv\n),\nranked AS (\n SELECT d.id, d.content, MIN(d.embedding <=> q.qv) AS best_distance\n FROM documents d, queries q\n GROUP BY d.id, d.content\n)\nSELECT * FROM ranked\nORDER BY best_distance\nLIMIT 20;\n```\n\n## Vector Arithmetic\n\nNQL supports vector arithmetic for "
|
|
},
|
|
{
|
|
"file": "pages/ops-backup.md",
|
|
"title": "Backup & Restore",
|
|
"section-id": "operations",
|
|
"keywords": "backup, restore, snapshot, WAL archiving, PITR, point-in-time recovery",
|
|
"description": "Backup and restore strategies for NeuralDB \u2014 snapshots, WAL archiving, and point-in-time recovery",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Backup & Restore\n\nA comprehensive backup strategy for NeuralDB combines base snapshots with continuous WAL archiving, enabling point-in-time recovery (PITR) to any moment within your retention window.\n\n## Backup Strategies\n\n| Strategy | Recovery point objective | Recovery time | Storage |\n|----------|--------------------------|---------------|---------|\n| Snapshot only | Time of last snapshot | Fast | Medium |\n| WAL archiving only | Continuous (any point) | Slow | High |\n| Snapshot + WAL | Best of both | Fast | High |\n\n**Recommendation:** Use snapshot + WAL archiving in production. Take daily base snapshots and archive WAL continuously.\n\n## Physical Snapshot (pg_basebackup)\n\n`pg_basebackup` creates a consistent physical copy of the data directory:\n\n```bash\n# Full backup \u2014 local filesystem\npg_basebackup \\\n --host=localhost \\\n --port=5432 \\\n --username=backup_user \\\n --pgdata=/backups/neuraldb/$(date +%Y%m%d) \\\n --wal-method=stream \\\n --checkpoint=fast \\\n --compress=lz4 \\\n --progress \\\n --verbose\n\n# Full backup \u2014 tar format (smaller, easier to upload to S3)\npg_basebackup \\\n --host=localhost \\\n --pgdata=- \\\n --format=tar \\\n --wal-method=stream \\\n --compress=lz4 \\\n | aws s3 cp - s3://my-backups/neuraldb/base-$(date +%Y%m%d).tar.lz4\n```\n\nCreate a dedicated backup user:\n\n```sql\nCREATE USER backup_user WITH REPLICATION PASSWORD 'backup-password';\nGRANT CONNECT ON DATABASE neuraldb TO backup_user;\n```\n\n## WAL Archiving\n\nWAL archiving copies each WAL segment to a secure location as it is completed. Combined with a base snapshot, this enables PITR.\n\nEnable WAL archiving:\n\n```ini\n# neuraldb.conf\nwal_level = replica\narchive_mode = on\narchive_command = 'aws s3 cp %p s3://my-backups/neuraldb/wal/%f'\narchive_timeout = 60 # archive at least every 60 seconds even if no WAL activity\n```\n\nVerify archiving is working:\n\n```sql\nSELECT last_archived_wal, last_archived_time,\n last_failed_wal, last_failed_time,\n archived_count, failed_count\nFROM pg_stat_archiver;\n```\n\n### S3 Archive Command\n\n```bash\n#!/bin/bash\n# /usr/local/bin/neuraldb-archive.sh\n# Usage: %p = source file path, %f = file name\n\nset -e\nSOURCE=\"$1\"\nDEST_FILE=\"$2\"\nS3_BUCKET=\"${ARCHIVE_S3_BUCKET}\"\nS3_PREFIX=\"${ARCHIVE_S3_PREFIX:-neuraldb/wal/}\"\n\naws s3 cp \"$SOURCE\" \"s3://${S3_BUCKET}/${S3_PREFIX}${DEST_FILE}\" \\\n --storage-class STANDARD_IA \\\n --sse aws:kms\n```\n\n```ini\narchive_command = '/usr/local/bin/neuraldb-archive.sh %p %f'\n```\n\n## Automated Backups with pgBackRest\n\npgBackRest is the recommended tool for production NeuralDB backups:\n\n```bash\n# Install\nsudo apt install pgbackrest\n\n# Configure\nsudo tee /etc/pgbackrest/pgbackrest.conf <<'EOF'\n[global]\nrepo1-path=/var/lib/pgbackrest\nrepo1-retention-full=7\nrepo1-retention-diff=14\nrepo1-type=s3\nrepo1-s3-bucket=my-neuraldb-backups\nrepo1-s3-endpoint=s3.amazonaws.com\nrepo1-s3-region=us-east-1\ncompress-type=lz4\nstart-fast=y\nbackup-standby=y\n\n[neuraldb]\npg1-path=/var/lib/neuraldb/data\npg1-port=5432\npg1-user=backup_user\nEOF\n\n# Initialise\nsudo -u postgres pgbackrest --stanza=neuraldb stanza-create\n\n# Full backup\nsudo -u postgres pgbackrest --stanza=neuraldb backup --type=full\n\n# Differential backup (only changes since last full)\nsudo -u postgres pgbackrest --stanza=neuraldb backup --type=diff\n\n# Incremental (only changes since last backup of any type)\nsudo -u postgres pgbackrest --stanza=neuraldb backup --type=incr\n```\n\nSchedule backups with cron:\n\n```cron\n# /etc/cron.d/neuraldb-backup\n0 1 * * 0 postgres pgbackrest --stanza=neuraldb backup --type=full\n0 1 * * 1-6 postgres pgbackrest --stanza=neuraldb backup --type=diff\n```\n\n## Point-in-Time Recovery (PITR)\n\nTo restore to a specific point in time:\n\n```bash\n# Stop NeuralDB\nsystemctl stop neuraldb\n\n# Restore a base backup\npgbackrest --stanza=neuraldb restore \\\n --target=\"2026-05-15 14:30:00+00\" \\\n --target-action=promote \\\n --delta\n\n# Or restore to just before a specific transaction\npgbackrest --stanza=neuraldb restore \\\n --target-name=\"before_accidental_delete\" \\\n --target-action=promote\n\n# Start NeuralDB \u2014 it will replay WAL up to the target point\nsystemctl start neuraldb\n```\n\nCreate named restore points before risky operations:\n\n```sql\n-- Before running a migration\nSELECT pg_create_restore_point('before_migration_20260515');\n```\n\n## Logical Backup (pg_dump)\n\nFor smaller databases or table-level backups, `pg_dump` provides a logical backup:\n\n```bash\n# Dump entire database\npg_dump -h localhost -U neuraldb mydb | \\\n lz4 | \\\n aws s3 cp - s3://my-backups/neuraldb/logical-$(date +%Y%m%d).sql.lz4\n\n# Dump specific table\npg_dump -h localhost -U neuraldb -t documents mydb > documents-backup.sql\n\n# Dump in custom format (best compression, selective restore)\npg_dump -Fc -h localhost -U neuraldb mydb > mydb-$(date +%Y%m%d).dump\n```\n\n**Note:** Logical backups do not include vector index data \u2014 only the raw vector column values. After restore, recreate indexes manually.\n\n## Restoring from pg_dump\n\n```bash\n# Restore entire database\nlz4 -d backup.sql.lz4"
|
|
},
|
|
{
|
|
"file": "pages/ops-migration.md",
|
|
"title": "Migration",
|
|
"section-id": "operations",
|
|
"keywords": "migration, import, Postgres, Pinecone, Weaviate, data migration, ETL",
|
|
"description": "Migrating data to NeuralDB from PostgreSQL, Pinecone, Weaviate, and other sources",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Migration\n\nThis guide covers migrating data into NeuralDB from common sources: PostgreSQL (with or without pgvector), Pinecone, and Weaviate.\n\n## From PostgreSQL (without vectors)\n\nIf you are migrating a standard PostgreSQL database to NeuralDB, the simplest path is a logical dump and restore:\n\n```bash\n# 1. Dump from source Postgres\npg_dump \\\n -h source-host \\\n -U source-user \\\n -d source-database \\\n --format=custom \\\n --compress=9 \\\n > source-backup.dump\n\n# 2. Create the target database in NeuralDB\npsql -h neuraldb-host -U neuraldb -c \"CREATE DATABASE myapp;\"\n\n# 3. Restore into NeuralDB\npg_restore \\\n -h neuraldb-host \\\n -U neuraldb \\\n -d myapp \\\n --jobs=8 \\\n --no-owner \\\n source-backup.dump\n```\n\n### Adding Vector Columns Post-Migration\n\nAfter restoring the schema and data, add vector columns and generate embeddings:\n\n```sql\n-- Add the vector column\nALTER TABLE documents ADD COLUMN embedding VECTOR(1536);\n\n-- Create the index (do this before backfilling on large tables)\nCREATE INDEX CONCURRENTLY documents_embedding_idx\nON documents USING hnsw (embedding vector_cosine_ops);\n```\n\nThen backfill embeddings in batches:\n\n```python\nimport openai\nfrom neuraldb import NeuralDB\n\nclient = NeuralDB(connection_string)\nopenai_client = openai.OpenAI()\n\nBATCH_SIZE = 100\n\nwhile True:\n rows = client.query(\"\"\"\n SELECT id, content FROM documents\n WHERE embedding IS NULL\n LIMIT %s\n \"\"\", [BATCH_SIZE])\n\n if not rows:\n break\n\n texts = [row['content'] for row in rows]\n response = openai_client.embeddings.create(\n model=\"text-embedding-3-small\",\n input=texts\n )\n\n updates = [\n (response.data[i].embedding, rows[i]['id'])\n for i in range(len(rows))\n ]\n\n client.executemany(\n \"UPDATE documents SET embedding = %s WHERE id = %s\",\n updates\n )\n print(f\"Backfilled {len(rows)} rows\")\n```\n\n## From PostgreSQL + pgvector\n\npgvector uses the same `VECTOR` type as NeuralDB. Migration is a direct dump and restore with minimal adjustments.\n\n```bash\n# Dump \u2014 exclude pgvector extension (NeuralDB has native vector support)\npg_dump \\\n -h source-host -U source-user -d source-db \\\n --format=custom \\\n --exclude-extension=vector \\\n > pgvector-backup.dump\n\npg_restore \\\n -h neuraldb-host -U neuraldb -d myapp \\\n --jobs=8 \\\n pgvector-backup.dump\n```\n\n### Re-create HNSW Indexes\n\npgvector HNSW indexes are not transferred. Recreate them in NeuralDB:\n\n```sql\n-- Drop pgvector-created indexes\nDROP INDEX IF EXISTS documents_embedding_idx;\n\n-- Create NeuralDB HNSW index (same syntax, better performance)\nCREATE INDEX CONCURRENTLY documents_embedding_idx\nON documents USING hnsw (embedding vector_cosine_ops)\nWITH (m = 16, ef_construction = 64);\n```\n\n## From Pinecone\n\nPinecone stores vectors with metadata. Export using the Pinecone SDK and ingest into NeuralDB:\n\n```python\nimport pinecone\nfrom neuraldb import NeuralDB, BulkIngestor\n\n# Source: Pinecone\npc = pinecone.Pinecone(api_key=os.environ[\"PINECONE_API_KEY\"])\nindex = pc.Index(\"my-index\")\n\n# Target: NeuralDB\nclient = NeuralDB(os.environ[\"NEURALDB_URL\"])\n\n# Create target table\nclient.execute(\"\"\"\n CREATE TABLE IF NOT EXISTS pinecone_migration (\n id TEXT PRIMARY KEY,\n embedding VECTOR(1536),\n metadata JSONB,\n migrated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()\n )\n\"\"\")\n\nclient.execute(\"\"\"\n CREATE INDEX IF NOT EXISTS pinecone_migration_emb_idx\n ON pinecone_migration USING hnsw (embedding vector_cosine_ops)\n\"\"\")\n\n# Paginate through all Pinecone vectors\ningestor = BulkIngestor(client, table=\"pinecone_migration\", batch_size=500)\n\nwith ingestor as ing:\n for ids_batch in paginate_pinecone_ids(index, batch_size=1000):\n fetch_response = index.fetch(ids=ids_batch)\n\n for vector_id, vector_data in fetch_response.vectors.items():\n ing.add({\n \"id\": vector_id,\n \"embedding\": vector_data.values,\n \"metadata\": vector_data.metadata or {}\n })\n\nprint(f\"Migrated {ingestor.total_inserted} vectors\")\n```\n\n### Mapping Pinecone Metadata to Columns\n\nFlatten commonly-queried metadata fields into dedicated columns for better query performance:\n\n```python\n# Instead of: metadata JSONB\n# Create typed columns for common filter fields:\nclient.execute(\"\"\"\n ALTER TABLE pinecone_migration\n ADD COLUMN IF NOT EXISTS category TEXT GENERATED ALWAYS AS (metadata->>'category') STORED,\n ADD COLUMN IF NOT EXISTS created_date DATE GENERATED ALWAYS AS ((metadata->>'date')::DATE) STORED;\n\n CREATE INDEX ON pinecone_migration (category);\n CREATE INDEX ON pinecone_migration (created_date);\n\"\"\")\n```\n\n## From Weaviate\n\nExport Weaviate data using the Weaviate client SDK:\n\n```python\nimport weaviate\nfrom neuraldb import NeuralDB, BulkIngestor\n\nweaviate_client = weaviate.connect_to_local()\nneuraldb_client = NeuralDB(os.environ[\"NEURALDB_URL\"])\n\ncollection = weaviate_client.collections.get(\"Document\")\n\n# Create target schema\nne"
|
|
},
|
|
{
|
|
"file": "pages/ops-monitoring.md",
|
|
"title": "Monitoring",
|
|
"section-id": "operations",
|
|
"keywords": "monitoring, Prometheus, Grafana, metrics, alerts, observability, dashboards",
|
|
"description": "Monitoring NeuralDB with Prometheus metrics, Grafana dashboards, and alert configuration",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Monitoring\n\n\n\nObservability is critical for database operations. NeuralDB exposes Prometheus-compatible metrics and provides an official Grafana dashboard for real-time monitoring.\n\n## Prometheus Metrics\n\nNeuralDB exposes metrics at `http://localhost:9187/metrics` (via the bundled exporter).\n\nEnable the metrics exporter:\n\n```ini\n# neuraldb.conf\nmetrics.enabled = true\nmetrics.port = 9187\nmetrics.path = /metrics\n```\n\nOr run the standalone exporter:\n\n```bash\nneuraldb_exporter \\\n --web.listen-address=:9187 \\\n --db.uri=\"postgresql://monitor:password@localhost:5432/neuraldb?sslmode=disable\"\n```\n\n### Key Metrics\n\n#### Connection Metrics\n\n| Metric | Type | Description |\n|--------|------|-------------|\n| `neuraldb_connections_total` | Gauge | Current connections by state |\n| `neuraldb_connections_max` | Gauge | `max_connections` setting |\n| `neuraldb_connection_pool_waiting` | Gauge | Queries waiting for a connection |\n\n#### Query Metrics\n\n| Metric | Type | Description |\n|--------|------|-------------|\n| `neuraldb_queries_total` | Counter | Total queries by database and status |\n| `neuraldb_query_duration_seconds` | Histogram | Query duration (p50, p95, p99) |\n| `neuraldb_slow_queries_total` | Counter | Queries exceeding `log_min_duration_statement` |\n| `neuraldb_deadlocks_total` | Counter | Deadlocks detected |\n\n#### Vector Metrics\n\n| Metric | Type | Description |\n|--------|------|-------------|\n| `neuraldb_vector_queries_total` | Counter | Vector similarity queries by index |\n| `neuraldb_vector_query_duration_seconds` | Histogram | ANN query latency |\n| `neuraldb_hnsw_index_size_bytes` | Gauge | In-memory size of HNSW graphs |\n| `neuraldb_hnsw_build_duration_seconds` | Histogram | Time to build HNSW indexes |\n| `neuraldb_vector_recall_ratio` | Gauge | Estimated recall for ANN queries |\n\n#### Replication Metrics\n\n| Metric | Type | Description |\n|--------|------|-------------|\n| `neuraldb_replication_lag_bytes` | Gauge | WAL lag per replica |\n| `neuraldb_replication_lag_seconds` | Gauge | Time lag per replica |\n| `neuraldb_wal_size_bytes` | Gauge | Current WAL on-disk size |\n\n#### Storage Metrics\n\n| Metric | Type | Description |\n|--------|------|-------------|\n| `neuraldb_database_size_bytes` | Gauge | Total database size |\n| `neuraldb_table_size_bytes` | Gauge | Size per table |\n| `neuraldb_bloat_ratio` | Gauge | Estimated dead row ratio |\n| `neuraldb_checkpoint_duration_seconds` | Histogram | Checkpoint write time |\n\n## Prometheus Configuration\n\n```yaml\n# prometheus.yml\nscrape_configs:\n - job_name: 'neuraldb'\n static_configs:\n - targets: ['localhost:9187']\n scrape_interval: 15s\n metrics_path: /metrics\n```\n\n## Grafana Dashboard\n\nImport the official NeuralDB dashboard from Grafana.com (Dashboard ID: **18921**):\n\n```bash\n# Import via Grafana API\ncurl -X POST \\\n http://admin:password@localhost:3000/api/dashboards/import \\\n -H \"Content-Type: application/json\" \\\n -d '{ \"gnetId\": 18921, \"overwrite\": true, \"inputs\": [{\"name\": \"DS_PROMETHEUS\", \"type\": \"datasource\", \"pluginId\": \"prometheus\", \"value\": \"Prometheus\"}] }'\n```\n\nThe dashboard includes panels for:\n- Query rate and error rate\n- Query latency percentiles (p50, p95, p99)\n- Active connections vs max connections\n- Vector index memory usage\n- Replication lag\n- Database and table sizes\n- Cache hit ratio\n- Checkpoint frequency\n\n## Alerting Rules\n\nCreate Prometheus alerting rules for critical conditions:\n\n```yaml\n# neuraldb-alerts.yml\ngroups:\n - name: neuraldb\n rules:\n\n - alert: NeuralDBConnectionsHigh\n expr: neuraldb_connections_total{state=\"active\"} / neuraldb_connections_max > 0.85\n for: 2m\n labels:\n severity: warning\n annotations:\n summary: \"NeuralDB connections above 85%\"\n description: \"{{ $value | humanizePercentage }} of max connections in use\"\n\n - alert: NeuralDBConnectionsExhausted\n expr: neuraldb_connections_total{state=\"active\"} / neuraldb_connections_max > 0.98\n for: 30s\n labels:\n severity: critical\n annotations:\n summary: \"NeuralDB connections nearly exhausted\"\n\n - alert: NeuralDBHighQueryLatency\n expr: histogram_quantile(0.99, rate(neuraldb_query_duration_seconds_bucket[5m])) > 1.0\n for: 5m\n labels:\n severity: warning\n annotations:\n summary: \"P99 query latency above 1 second\"\n\n - alert: NeuralDBReplicationLagHigh\n expr: neuraldb_replication_lag_seconds > 30\n for: 1m\n labels:\n severity: warning\n annotations:\n summary: \"Replication lag above 30 seconds\"\n\n - alert: NeuralDBDiskSpaceHigh\n expr: (neuraldb_database_size_bytes / disk_total_bytes) > 0.80\n for: 5m\n labels:\n severity: warning\n annotations:\n summary: \"Database storage above 80% capacity\"\n\n - alert: NeuralDBVectorBufferExhausted\n expr: neuraldb_hnsw_index_siz"
|
|
},
|
|
{
|
|
"file": "pages/ops-scaling.md",
|
|
"title": "Scaling",
|
|
"section-id": "operations",
|
|
"keywords": "scaling, sharding, read replicas, horizontal scaling, capacity planning, performance",
|
|
"description": "Scaling NeuralDB horizontally with sharding, read replicas, and capacity planning",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Scaling\n\nNeuralDB is designed to scale horizontally. This page covers adding read replicas for query throughput, sharding for data volume, and capacity planning to avoid resource exhaustion.\n\n## Vertical Scaling (Scale Up)\n\nBefore adding nodes, ensure you have maximised single-node performance:\n\n### Memory\n\nThe biggest lever for NeuralDB performance is memory. Ensure:\n- `vector_buffer` is large enough to hold all active HNSW graphs\n- `shared_buffers` is set to 25% of RAM\n- `work_mem` is appropriate for your query patterns\n\n```sql\n-- Check if vectors are being served from disk (slow) vs memory (fast)\nSELECT index_name, hnsw_graph_size_bytes, hnsw_in_memory\nFROM neuraldb_stat_vector_indexes\nORDER BY hnsw_graph_size_bytes DESC;\n```\n\nIf `hnsw_in_memory = false`, increase `vector_buffer`.\n\n### CPU\n\nVector ANN searches are CPU-bound. Enable parallel query:\n\n```ini\nmax_parallel_workers_per_gather = 8\nmax_parallel_workers = 16\n```\n\n```sql\n-- Allow parallel ANN queries for large tables\nSET max_parallel_workers_per_gather = 8;\nSELECT * FROM large_table ORDER BY embedding <=> $1 LIMIT 10;\n```\n\n### Storage I/O\n\nUse NVMe SSDs with high IOPS. Configure the OS:\n\n```bash\n# Increase read-ahead for sequential I/O\nsudo blockdev --setra 1024 /dev/nvme0n1\n\n# Use deadline/mq-deadline I/O scheduler\necho \"mq-deadline\" | sudo tee /sys/block/nvme0n1/queue/scheduler\n\n# Disable transparent huge pages (reduces latency variability)\necho never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled\n```\n\n## Read Replicas\n\nAdd read replicas to distribute query load.\n\n### Setting Up Read Replicas\n\nFollow the [Replication guide](config-replication.md) to add replicas. Each replica can independently serve `SELECT` queries, including vector similarity searches.\n\n### Client-Side Read Splitting\n\nConfigure your application to route reads to replicas:\n\n**Python:**\n```python\nfrom neuraldb import NeuralDB\n\nprimary = NeuralDB(\"postgresql://neuraldb:pass@primary:5432/mydb\")\nreplica = NeuralDB(\"postgresql://neuraldb:pass@replica:5432/mydb\")\n\ndef search(query_vector):\n # Read goes to replica\n return replica.query(\"SELECT * FROM docs ORDER BY embedding <=> %s LIMIT 10\", [query_vector])\n\ndef insert(content, embedding):\n # Write goes to primary\n return primary.execute(\"INSERT INTO docs (content, embedding) VALUES (%s, %s)\", [content, embedding])\n```\n\n**Connection string with `target_session_attrs`:**\n```\npostgresql://neuraldb:pass@primary:5432,replica:5432/mydb?target_session_attrs=prefer-standby\n```\n\n### Read Replica Scaling Targets\n\n| Replicas | Approximate peak QPS (1536-dim, 10M vectors) |\n|---------|----------------------------------------------|\n| 1 primary | 8,000 |\n| 1 primary + 2 replicas | 24,000 |\n| 1 primary + 4 replicas | 48,000 |\n| 1 primary + 8 replicas | 96,000 |\n\n## Horizontal Sharding\n\nFor datasets exceeding single-node capacity (>50M vectors or >5 TB), shard across multiple primary nodes.\n\n### Shard Configuration\n\n```sql\n-- Create a sharded cluster (requires NeuralDB Cluster Edition)\nSELECT neuraldb_cluster.init_cluster(\n shards => 8,\n replication_factor => 2\n);\n\n-- Create a sharded table\nCREATE TABLE documents (\n id UUID NOT NULL DEFAULT gen_random_uuid(),\n tenant_id UUID NOT NULL,\n content TEXT,\n embedding VECTOR(1536)\n) SHARD BY tenant_id;\n\n-- Each shard holds ~1/8 of the data\n-- All rows with the same tenant_id are colocated on the same shard\n```\n\n### Cross-Shard Queries\n\nCross-shard queries (where the filter doesn't align with the shard key) are automatically parallelised across shards:\n\n```sql\n-- This query executes on all 8 shards in parallel\nSELECT id, content, 1 - (embedding <=> $1) AS similarity\nFROM documents\nORDER BY embedding <=> $1\nLIMIT 10;\n-- Results are merged and re-ranked by the coordinator\n```\n\nPerformance with 8 shards: near-linear scaling. An 8-shard cluster serves ~8\u00d7 the QPS of a single node for cross-shard searches, with ~20% overhead for coordination.\n\n### Shard Rebalancing\n\nWhen adding new shard nodes, rebalance data:\n\n```sql\n-- Rebalance shards (online, non-blocking)\nSELECT neuraldb_cluster.rebalance_shards();\n\n-- Monitor progress\nSELECT * FROM neuraldb_cluster.rebalance_status;\n```\n\n## Capacity Planning\n\n### Storage Capacity\n\nEstimate required storage:\n\n```\nRow data \u2248 avg_row_size_bytes \u00d7 num_rows \u00d7 1.3 (index overhead)\nVector data \u2248 dimensions \u00d7 4 bytes \u00d7 num_vectors\nHNSW graph \u2248 dimensions \u00d7 4 bytes \u00d7 num_vectors \u00d7 1.3\nWAL \u2248 daily_write_volume \u00d7 wal_retention_days\n\nTotal \u2248 row_data + vector_data + HNSW_graph + WAL + 20% buffer\n```\n\nExample: 100M rows, 1536 dimensions, 500 bytes average row size:\n- Row data: 500B \u00d7 100M \u00d7 1.3 \u2248 **65 GB**\n- Vector data: 1536 \u00d7 4B \u00d7 100M \u2248 **614 GB**\n- HNSW graph: 614 GB \u00d7 1.3 \u2248 **800 GB** (must fit in `vector_buffer`)\n- WAL (7 days): 10 GB/day \u00d7 7 = **70 GB**\n- **Total: ~1.6 TB storage, 800 GB RAM for HNSW**\n\n### Connection Capacity\n\n```\nmax_connections = max_app_connections + pgbouncer_pool_size + replication_slots + 3 (superuser)\n```\n\nFor 500 app connecti"
|
|
},
|
|
{
|
|
"file": "pages/ops-troubleshooting.md",
|
|
"title": "Troubleshooting",
|
|
"section-id": "operations",
|
|
"keywords": "troubleshooting, errors, diagnostics, FAQ, common problems, debug",
|
|
"description": "Common NeuralDB errors, diagnostic techniques, and frequently asked questions",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Troubleshooting\n\nThis page covers common NeuralDB errors and how to diagnose and resolve them.\n\n## Connection Issues\n\n### `FATAL: password authentication failed for user \"neuraldb\"`\n\nThe password is incorrect or the user doesn't exist.\n\n```bash\n# Reset the password as the OS postgres user\nsudo -u neuraldb neuraldb-cli\n```\n\n```sql\nALTER USER neuraldb PASSWORD 'new-password';\n```\n\nCheck `pg_hba.conf` \u2014 ensure the correct authentication method is used for the client's IP address.\n\n### `FATAL: no pg_hba.conf entry for host \"x.x.x.x\", user \"neuraldb\"`\n\nThe client's IP is not in `pg_hba.conf`:\n\n```\n# Add to pg_hba.conf\nhost all all x.x.x.x/32 scram-sha-256\n```\n\nReload: `SELECT pg_reload_conf();`\n\n### `could not connect to server: Connection refused`\n\nNeuralDB is not running on the expected host/port:\n\n```bash\n# Check process\nsystemctl status neuraldb\n# or\nps aux | grep neuraldb\n\n# Check listening port\nss -tlnp | grep 5432\n\n# Check logs\njournalctl -u neuraldb -n 50\n```\n\n### `FATAL: remaining connection slots are reserved for non-replication superuser connections`\n\nAll available connections are consumed. Use PgBouncer to pool connections, or increase `max_connections`:\n\n```sql\n-- Check current connections\nSELECT count(*), state, wait_event_type FROM pg_stat_activity GROUP BY state, wait_event_type;\n\n-- Kill idle connections\nSELECT pg_terminate_backend(pid) FROM pg_stat_activity\nWHERE state = 'idle' AND state_change < NOW() - INTERVAL '10 minutes';\n```\n\n## Vector Query Issues\n\n### Slow Vector Searches\n\nIf vector queries are slow, check whether the HNSW index is being used:\n\n```sql\nEXPLAIN (ANALYZE, BUFFERS)\nSELECT id, embedding <=> '[...]' AS dist\nFROM documents\nORDER BY embedding <=> '[...]'\nLIMIT 10;\n```\n\nLook for `Index Scan using documents_embedding_idx` in the plan. If you see `Seq Scan`, the planner may have decided the index is not beneficial.\n\nCommon causes:\n1. **Missing LIMIT clause**: The planner only uses the HNSW index for `ORDER BY ... LIMIT` queries.\n2. **Too few rows**: For small tables, a sequential scan may be faster.\n3. **HNSW graph not in memory**: Check `SELECT * FROM neuraldb_stat_vector_indexes` \u2014 if `hnsw_in_memory = false`, increase `vector_buffer`.\n4. **ef_search too low**: Increase for better recall at the cost of speed.\n\n```sql\n-- Force index use for debugging\nSET enable_seqscan = off;\nEXPLAIN ANALYZE SELECT ... ORDER BY embedding <=> $1 LIMIT 10;\nSET enable_seqscan = on;\n```\n\n### `ERROR: expected 1536 dimensions, not 768`\n\nThe vector you are inserting has a different number of dimensions than the column definition:\n\n```sql\n-- Check column definition\n\\d documents\n-- embedding column shows VECTOR(1536)\n\n-- You are inserting a 768-dimensional vector \u2014 check your embedding model\n```\n\nEnsure your embedding model is consistent. If you need to change models, you must re-embed all existing data.\n\n### Low Recall on ANN Queries\n\nIf approximate queries are not returning expected results:\n\n```sql\n-- Increase ef_search for higher recall\nSET hnsw.ef_search = 200;\nSELECT id, content, 1 - (embedding <=> $1) AS similarity\nFROM documents\nORDER BY embedding <=> $1\nLIMIT 10;\n```\n\nCompare against exact search:\n\n```sql\nSET neuraldb.vector_scan = 'exact';\nSELECT id, content FROM documents ORDER BY embedding <=> $1 LIMIT 10;\n```\n\nIf exact search finds results that approximate search misses, increase `ef_search` or rebuild the index with a higher `m` value.\n\n## Performance Issues\n\n### High Memory Usage\n\n```sql\n-- Check vector index memory consumption\nSELECT index_name, pg_size_pretty(hnsw_graph_size_bytes) AS graph_memory\nFROM neuraldb_stat_vector_indexes\nORDER BY hnsw_graph_size_bytes DESC;\n\n-- Check for shared_buffers usage\nSELECT name, setting, unit FROM pg_settings\nWHERE name IN ('shared_buffers', 'vector_buffer', 'work_mem');\n```\n\n### Disk Space Exhaustion\n\n```sql\n-- Identify large tables and indexes\nSELECT tablename, pg_size_pretty(pg_total_relation_size(tablename::regclass)) AS size\nFROM pg_tables WHERE schemaname = 'public'\nORDER BY pg_total_relation_size(tablename::regclass) DESC\nLIMIT 20;\n\n-- Check WAL accumulation (often caused by idle replication slots)\nSELECT slot_name, active, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS lag\nFROM pg_replication_slots\nORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC;\n```\n\nDrop inactive replication slots:\n\n```sql\nSELECT pg_drop_replication_slot('orphaned_slot_name');\n```\n\n### Long-Running Queries\n\n```sql\n-- Find queries running longer than 30 seconds\nSELECT pid, now() - pg_stat_activity.query_start AS duration, query, state\nFROM pg_stat_activity\nWHERE state != 'idle'\n AND (now() - pg_stat_activity.query_start) > INTERVAL '30 seconds'\nORDER BY duration DESC;\n\n-- Terminate a specific query\nSELECT pg_cancel_backend(pid); -- send SIGINT (graceful)\nSELECT pg_terminate_backend(pid); -- send SIGTERM (forceful)\n```\n\n## Replication Issues\n\n### Replication Lag Growing\n\n```sql\n-- Check lag on primary\nSELECT client_addr, state, s"
|
|
},
|
|
{
|
|
"file": "pages/sdk-go.md",
|
|
"title": "Go SDK",
|
|
"section-id": "client-sdks",
|
|
"keywords": "Go, Golang, SDK, client, connection pool, query builder, pgx",
|
|
"description": "The NeuralDB Go SDK \u2014 installation, connection pooling, and vector query builder",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Go SDK\n\nThe NeuralDB Go SDK provides idiomatic Go bindings built on top of `pgx`, the high-performance PostgreSQL driver for Go.\n\n## Installation\n\n```bash\ngo get github.com/neuraldb/neuraldb-go\n```\n\nRequires Go 1.21+.\n\n## Connecting\n\n### Single Connection\n\n```go\npackage main\n\nimport (\n \"context\"\n \"fmt\"\n \"log\"\n\n \"github.com/neuraldb/neuraldb-go\"\n)\n\nfunc main() {\n ctx := context.Background()\n\n client, err := neuraldb.Connect(ctx, \"postgresql://neuraldb:password@localhost:5432/mydb\")\n if err != nil {\n log.Fatal(\"connection failed:\", err)\n }\n defer client.Close(ctx)\n\n var count int64\n err = client.QueryRow(ctx, \"SELECT COUNT(*) FROM documents\").Scan(&count)\n if err != nil {\n log.Fatal(err)\n }\n fmt.Println(\"Documents:\", count)\n}\n```\n\n### Connection Pool (Recommended)\n\n```go\nimport (\n \"github.com/neuraldb/neuraldb-go\"\n \"github.com/jackc/pgx/v5/pgxpool\"\n)\n\nfunc NewPool(ctx context.Context) (*neuraldb.Pool, error) {\n config, err := pgxpool.ParseConfig(os.Getenv(\"NEURALDB_URL\"))\n if err != nil {\n return nil, err\n }\n\n config.MaxConns = 20\n config.MinConns = 5\n config.MaxConnIdleTime = 30 * time.Minute\n config.MaxConnLifetime = time.Hour\n\n return neuraldb.NewPool(ctx, config)\n}\n```\n\n## Working with Vectors\n\n### Defining Vector Types\n\n```go\npackage main\n\nimport \"github.com/neuraldb/neuraldb-go/types\"\n\n// Create a vector from a float32 slice\nv := types.NewVector([]float32{0.023, -0.187, 0.412})\n\n// Create from float64 (auto-converted)\nv2 := types.NewVectorFromFloat64([]float64{0.023, -0.187, 0.412})\n\n// Access the underlying data\nfloats := v.Slice() // []float32\ndims := v.Dims() // int\n```\n\n### Inserting a Document with a Vector\n\n```go\ntype Document struct {\n ID string\n Content string\n Embedding types.Vector\n}\n\nfunc InsertDocument(ctx context.Context, pool *neuraldb.Pool, doc Document) error {\n _, err := pool.Exec(ctx,\n `INSERT INTO documents (id, content, embedding) VALUES ($1, $2, $3)`,\n doc.ID, doc.Content, doc.Embedding,\n )\n return err\n}\n```\n\n### Vector Similarity Search\n\n```go\nfunc SemanticSearch(ctx context.Context, pool *neuraldb.Pool, queryEmbedding []float32, limit int) ([]SearchResult, error) {\n qv := types.NewVector(queryEmbedding)\n\n rows, err := pool.Query(ctx, `\n SELECT id, content, 1 - (embedding <=> $1) AS similarity\n FROM documents\n WHERE embedding IS NOT NULL\n ORDER BY embedding <=> $1\n LIMIT $2\n `, qv, limit)\n if err != nil {\n return nil, fmt.Errorf(\"query failed: %w\", err)\n }\n defer rows.Close()\n\n var results []SearchResult\n for rows.Next() {\n var r SearchResult\n err := rows.Scan(&r.ID, &r.Content, &r.Similarity)\n if err != nil {\n return nil, err\n }\n results = append(results, r)\n }\n\n return results, rows.Err()\n}\n```\n\n## Query Builder\n\nThe SDK includes an optional query builder for type-safe query construction:\n\n```go\nimport \"github.com/neuraldb/neuraldb-go/qb\"\n\n// Build a hybrid query\nquery, args := qb.New().\n Select(\"id\", \"name\", \"price\").\n Expr(\"1 - (embedding <=> $?) AS similarity\", queryVector).\n From(\"products\").\n Where(qb.Eq(\"category\", \"electronics\")).\n Where(qb.GTE(\"price\", 50)).\n Where(qb.LTE(\"price\", 500)).\n Where(qb.GT(\"stock\", 0)).\n OrderByExpr(\"embedding <=> $?\", queryVector).\n Limit(20).\n Build()\n\nrows, err := pool.Query(ctx, query, args...)\n```\n\n## Batch Operations\n\n```go\nimport \"github.com/jackc/pgx/v5\"\n\nfunc BatchInsert(ctx context.Context, pool *neuraldb.Pool, docs []Document) error {\n batch := &pgx.Batch{}\n\n for _, doc := range docs {\n batch.Queue(\n `INSERT INTO documents (content, embedding) VALUES ($1, $2)`,\n doc.Content, doc.Embedding,\n )\n }\n\n results := pool.SendBatch(ctx, batch)\n defer results.Close()\n\n for range docs {\n _, err := results.Exec()\n if err != nil {\n return fmt.Errorf(\"batch insert error: %w\", err)\n }\n }\n\n return results.Close()\n}\n```\n\n## Transactions\n\n```go\nfunc TransactionalInsert(ctx context.Context, pool *neuraldb.Pool, docs []Document) error {\n return pool.BeginTxFunc(ctx, pgx.TxOptions{\n IsoLevel: pgx.ReadCommitted,\n }, func(tx pgx.Tx) error {\n for _, doc := range docs {\n _, err := tx.Exec(ctx,\n `INSERT INTO documents (content, embedding) VALUES ($1, $2)`,\n doc.Content, doc.Embedding,\n )\n if err != nil {\n return err // auto-rolled back by BeginTxFunc\n }\n }\n\n _, err := tx.Exec(ctx,\n `UPDATE stats SET doc_count = doc_count + $1`,\n len(docs),\n )\n return err\n })\n}\n```\n\n## Scanning Results into Structs\n\n```go\nimport \"github.com/jackc/pgx/v5/pgxscan\"\n\ntype Product struct {\n ID string `db:\"id\"`\n Name strin"
|
|
},
|
|
{
|
|
"file": "pages/sdk-javascript.md",
|
|
"title": "JavaScript SDK",
|
|
"section-id": "client-sdks",
|
|
"keywords": "JavaScript, TypeScript, SDK, Node.js, browser, npm, client",
|
|
"description": "The NeuralDB JavaScript/TypeScript SDK for Node.js and browser environments",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# JavaScript SDK\n\nThe NeuralDB JavaScript SDK provides a fully typed client for Node.js and browser environments. It is built on `pg` (node-postgres) for Node.js and a custom HTTP adapter for edge and browser environments.\n\n## Installation\n\n```bash\nnpm install @neuraldb/client\n# or\nyarn add @neuraldb/client\n# or\npnpm add @neuraldb/client\n```\n\n## Basic Setup\n\n### Node.js\n\n```typescript\nimport { NeuralDB } from '@neuraldb/client';\n\nconst client = new NeuralDB({\n connectionString: process.env.NEURALDB_URL!,\n // or individual options:\n host: 'localhost',\n port: 5432,\n user: 'neuraldb',\n password: 'password',\n database: 'mydb',\n ssl: { rejectUnauthorized: true },\n});\n\nawait client.connect();\n```\n\n### Edge / Serverless (HTTP mode)\n\nFor Cloudflare Workers, Vercel Edge, and browser environments, use the HTTP adapter:\n\n```typescript\nimport { NeuralDB, HttpAdapter } from '@neuraldb/client';\n\nconst client = new NeuralDB({\n adapter: new HttpAdapter({\n url: process.env.NEURALDB_HTTP_URL!,\n apiKey: process.env.NEURALDB_API_KEY!,\n }),\n});\n```\n\n### Connection Pool\n\n```typescript\nimport { NeuralDBPool } from '@neuraldb/client';\n\nconst pool = new NeuralDBPool({\n connectionString: process.env.NEURALDB_URL!,\n max: 20,\n idleTimeoutMillis: 30000,\n connectionTimeoutMillis: 5000,\n});\n```\n\n## Executing Queries\n\n```typescript\n// Simple query \u2014 returns QueryResult\nconst result = await client.query('SELECT id, content FROM documents LIMIT 10');\nconst rows = result.rows; // typed as any[]\n\n// Parameterised query\nconst { rows } = await client.query<{ id: string; content: string }>(\n 'SELECT id, content FROM documents WHERE source = $1 LIMIT $2',\n ['web-scraper', 20]\n);\n```\n\n## Vector Operations\n\n### Inserting Vectors\n\n```typescript\nimport { toVector } from '@neuraldb/client';\n\nconst embedding = [0.023, -0.187, 0.412, /* 1536 values */];\n\nawait client.query(\n 'INSERT INTO documents (content, embedding) VALUES ($1, $2)',\n ['My document content', toVector(embedding)]\n);\n```\n\n### Similarity Search\n\n```typescript\nimport OpenAI from 'openai';\nimport { toVector } from '@neuraldb/client';\n\nconst openai = new OpenAI();\n\nasync function semanticSearch(query: string, limit = 10) {\n const embeddingResponse = await openai.embeddings.create({\n model: 'text-embedding-3-small',\n input: query,\n });\n\n const queryVector = embeddingResponse.data[0].embedding;\n\n const { rows } = await client.query<{\n id: string;\n content: string;\n similarity: number;\n }>(\n `SELECT id, content, 1 - (embedding <=> $1) AS similarity\n FROM documents\n WHERE embedding IS NOT NULL\n ORDER BY embedding <=> $1\n LIMIT $2`,\n [toVector(queryVector), limit]\n );\n\n return rows;\n}\n```\n\n### Hybrid Search\n\n```typescript\nasync function hybridSearch(query: string, filters: Record<string, unknown>, limit = 10) {\n const queryVector = await generateEmbedding(query);\n\n const conditions: string[] = [];\n const params: unknown[] = [toVector(queryVector)];\n\n Object.entries(filters).forEach(([key, value]) => {\n params.push(value);\n conditions.push(`${key} = $${params.length}`);\n });\n\n const whereClause = conditions.length > 0\n ? 'WHERE ' + conditions.join(' AND ')\n : '';\n\n params.push(limit);\n\n const { rows } = await client.query(\n `SELECT id, content, 1 - (embedding <=> $1) AS similarity\n FROM documents\n ${whereClause}\n ORDER BY embedding <=> $1\n LIMIT $${params.length}`,\n params\n );\n\n return rows;\n}\n```\n\n## High-Level Document API\n\nThe SDK includes a higher-level document management API:\n\n```typescript\nimport { DocumentStore } from '@neuraldb/client';\n\nconst store = new DocumentStore(client, {\n table: 'documents',\n embeddingColumn: 'embedding',\n contentColumn: 'content',\n embeddingModel: {\n provider: 'openai',\n model: 'text-embedding-3-small',\n apiKey: process.env.OPENAI_API_KEY!,\n },\n});\n\n// Add documents (auto-generates embeddings)\nawait store.add([\n { content: 'First document', metadata: { source: 'web' } },\n { content: 'Second document', metadata: { source: 'pdf' } },\n]);\n\n// Search\nconst results = await store.search('query text', {\n limit: 10,\n filter: { source: 'web' },\n minSimilarity: 0.7,\n});\n\n// Delete\nawait store.delete({ filter: { source: 'web' } });\n```\n\n## Transactions\n\n```typescript\nconst pgClient = await pool.connect();\n\ntry {\n await pgClient.query('BEGIN');\n\n await pgClient.query(\n 'INSERT INTO documents (content, embedding) VALUES ($1, $2)',\n ['Content', toVector(embedding)]\n );\n\n await pgClient.query(\n 'UPDATE stats SET doc_count = doc_count + 1 WHERE id = $1',\n [statsId]\n );\n\n await pgClient.query('COMMIT');\n} catch (error) {\n await pgClient.query('ROLLBACK');\n throw error;\n} finally {\n pgClient.release();\n}\n```\n\n## Streaming Results\n\nFor large result sets, stream rows to avoid loading all data into memory:\n\n```typescript\nconst stream = client.queryStream(\n 'SELECT id, content, embedding FROM documents WHERE source = $1',\n ['web-sc"
|
|
},
|
|
{
|
|
"file": "pages/sdk-python.md",
|
|
"title": "Python SDK",
|
|
"section-id": "client-sdks",
|
|
"keywords": "Python, SDK, client, connection, CRUD, vector operations, psycopg",
|
|
"description": "Installing and using the NeuralDB Python SDK \u2014 connection, CRUD, and vector operations",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# Python SDK\n\nThe NeuralDB Python SDK provides a high-level client for Python applications. It is built on top of `psycopg3` (the PostgreSQL adapter) with NeuralDB-specific helpers for vector operations, embedding generation, and batch ingestion.\n\n## Installation\n\n```bash\npip install neuraldb\n# or\npip install neuraldb[asyncio] # includes async support\npip install neuraldb[all] # includes all optional extras\n```\n\n### Requirements\n\n- Python 3.10+\n- libpq (PostgreSQL client library)\n\nOn Ubuntu: `sudo apt install libpq-dev`\nOn macOS: `brew install libpq`\n\n## Connecting\n\n### Synchronous Client\n\n```python\nfrom neuraldb import NeuralDB\n\n# From connection string\nclient = NeuralDB(\"postgresql://neuraldb:password@localhost:5432/mydb\")\n\n# From parameters\nclient = NeuralDB(\n host=\"localhost\",\n port=5432,\n user=\"neuraldb\",\n password=\"password\",\n database=\"mydb\",\n sslmode=\"require\",\n)\n\n# Context manager (auto-closes connection)\nwith NeuralDB(\"postgresql://...\") as client:\n result = client.query(\"SELECT 1\")\n```\n\n### Async Client\n\n```python\nimport asyncio\nfrom neuraldb import AsyncNeuralDB\n\nasync def main():\n async with AsyncNeuralDB(\"postgresql://neuraldb:password@localhost/mydb\") as client:\n result = await client.query(\"SELECT 1\")\n print(result)\n\nasyncio.run(main())\n```\n\n### Connection Pool\n\n```python\nfrom neuraldb import NeuralDBPool\n\npool = NeuralDBPool(\n \"postgresql://neuraldb:password@localhost/mydb\",\n min_size=5,\n max_size=20,\n)\n\nwith pool.acquire() as client:\n result = client.query(\"SELECT COUNT(*) FROM documents\")\n```\n\n## Schema Operations\n\n```python\n# Create a table with a vector column\nclient.execute(\"\"\"\n CREATE TABLE IF NOT EXISTS documents (\n id UUID PRIMARY KEY DEFAULT gen_random_uuid(),\n content TEXT NOT NULL,\n source TEXT,\n embedding VECTOR(1536),\n created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()\n )\n\"\"\")\n\n# Create a vector index\nclient.execute(\"\"\"\n CREATE INDEX IF NOT EXISTS documents_embedding_idx\n ON documents USING hnsw (embedding vector_cosine_ops)\n WITH (m = 16, ef_construction = 64)\n\"\"\")\n```\n\n## CRUD Operations\n\n### Insert\n\n```python\nfrom neuraldb import Vector\n\n# Insert with pre-computed embedding\nclient.execute(\n \"INSERT INTO documents (content, source, embedding) VALUES (%s, %s, %s)\",\n (\"My document content\", \"web-scraper\", Vector([0.023, -0.187, 0.412, ...]))\n)\n\n# Insert many (batched for efficiency)\ndocs = [\n (\"Content A\", \"source-1\", Vector([...])),\n (\"Content B\", \"source-2\", Vector([...])),\n (\"Content C\", \"source-1\", Vector([...])),\n]\nclient.executemany(\n \"INSERT INTO documents (content, source, embedding) VALUES (%s, %s, %s)\",\n docs\n)\n```\n\n### Query\n\n```python\n# Standard query \u2014 returns list of Row objects\nrows = client.query(\"SELECT id, content FROM documents WHERE source = %s\", (\"web-scraper\",))\n\nfor row in rows:\n print(row[\"id\"], row[\"content\"])\n\n# As dicts\nrows = client.query(\n \"SELECT * FROM documents LIMIT 10\",\n row_factory=\"dict\"\n)\n\n# As named tuples\nrows = client.query(\n \"SELECT id, content FROM documents LIMIT 10\",\n row_factory=\"namedtuple\"\n)\n```\n\n### Vector Search\n\n```python\nimport openai\n\n# Generate query embedding\nquery_text = \"high-performance wireless headphones\"\nquery_embedding = openai.embeddings.create(\n model=\"text-embedding-3-small\",\n input=query_text\n).data[0].embedding\n\n# Semantic search\nresults = client.query(\"\"\"\n SELECT id, content, 1 - (embedding <=> %s) AS similarity\n FROM documents\n WHERE embedding IS NOT NULL\n ORDER BY embedding <=> %s\n LIMIT 10\n\"\"\", (Vector(query_embedding), Vector(query_embedding)))\n\nfor row in results:\n print(f\"{row['similarity']:.3f}: {row['content'][:100]}\")\n```\n\n### Using the High-Level Search API\n\n```python\nfrom neuraldb import VectorSearch\n\nsearcher = VectorSearch(client, table=\"documents\", embedding_column=\"embedding\")\n\nresults = searcher.search(\n query_vector=query_embedding,\n limit=10,\n filters={\"source\": \"web-scraper\"},\n metric=\"cosine\",\n)\n```\n\n### Update\n\n```python\nclient.execute(\n \"UPDATE documents SET content = %s, embedding = %s WHERE id = %s\",\n (\"Updated content\", Vector(new_embedding), doc_id)\n)\n```\n\n### Delete\n\n```python\nclient.execute(\"DELETE FROM documents WHERE id = %s\", (doc_id,))\n```\n\n## Transactions\n\n```python\nwith client.transaction():\n client.execute(\"INSERT INTO documents (content, embedding) VALUES (%s, %s)\", (content, Vector(embedding)))\n client.execute(\"UPDATE stats SET count = count + 1\")\n # Auto-commits on exit, rolls back on exception\n```\n\nExplicit control:\n\n```python\nwith client.transaction() as txn:\n try:\n client.execute(\"INSERT ...\")\n client.execute(\"UPDATE ...\")\n txn.commit()\n except Exception:\n txn.rollback()\n raise\n```\n\n## Bulk Ingestion\n\nFor high-throughput ingestion, use the `BulkIngestor`:\n\n```python\nfrom neuraldb import BulkIngestor\n\ningestor = BulkIngestor(\n client,\n "
|
|
},
|
|
{
|
|
"file": "pages/sdk-rest.md",
|
|
"title": "REST API",
|
|
"section-id": "client-sdks",
|
|
"keywords": "REST API, HTTP, endpoints, authentication, JSON, API",
|
|
"description": "NeuralDB REST API reference \u2014 all endpoints, authentication headers, and response formats",
|
|
"author": null,
|
|
"date": "",
|
|
"datetime": "",
|
|
"language": "en",
|
|
"body": "# REST API\n\nNeuralDB provides an HTTP REST API for environments where a direct database connection is not practical (browser clients, third-party integrations, webhooks). The REST API is a thin HTTP wrapper over NQL.\n\n## Base URL\n\n```\nhttps://your-neuraldb-host:8080/api/v1\n```\n\nFor NeuralDB Cloud:\n\n```\nhttps://[cluster-id].cloud.neuraldb.io/api/v1\n```\n\n## Authentication\n\nAll REST API requests require an API key in the `Authorization` header:\n\n```\nAuthorization: Bearer ndb_live_your_api_key_here\n```\n\nOr as a query parameter (less secure, avoid in production):\n\n```\n?api_key=ndb_live_your_api_key_here\n```\n\nCreate API keys in the NeuralDB CLI or the Cloud dashboard:\n\n```bash\nneuraldb-cli -c \"SELECT neuraldb_apikeys.create_key(label => 'my-app', role => 'app_readwrite')\"\n```\n\n## Query Endpoint\n\nExecute any NQL query:\n\n### `POST /query`\n\n```http\nPOST /api/v1/query\nContent-Type: application/json\nAuthorization: Bearer ndb_live_...\n\n{\n \"query\": \"SELECT id, content, 1 - (embedding <=> $1) AS similarity FROM documents ORDER BY embedding <=> $1 LIMIT 5\",\n \"params\": [[0.023, -0.187, 0.412]],\n \"database\": \"mydb\"\n}\n```\n\n**Response:**\n\n```json\n{\n \"rows\": [\n { \"id\": \"uuid-1\", \"content\": \"First document\", \"similarity\": 0.923 },\n { \"id\": \"uuid-2\", \"content\": \"Second document\", \"similarity\": 0.891 }\n ],\n \"rowCount\": 2,\n \"fields\": [\n { \"name\": \"id\", \"dataTypeID\": 2950 },\n { \"name\": \"content\", \"dataTypeID\": 25 },\n { \"name\": \"similarity\", \"dataTypeID\": 701 }\n ],\n \"executionTimeMs\": 3.2\n}\n```\n\n### Query Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `query` | string | Yes | NQL query string |\n| `params` | array | No | Query parameters (replaces $1, $2, ...) |\n| `database` | string | No | Target database (default: `neuraldb`) |\n| `timeout_ms` | integer | No | Query timeout in milliseconds |\n| `explain` | boolean | No | Return query execution plan |\n\n## Document Endpoints\n\nHigher-level CRUD endpoints for document management:\n\n### `POST /collections/:collection/documents`\n\nInsert documents (auto-generates embeddings if configured):\n\n```http\nPOST /api/v1/collections/my_docs/documents\nContent-Type: application/json\nAuthorization: Bearer ndb_live_...\n\n{\n \"documents\": [\n {\n \"content\": \"NeuralDB is an AI-native database\",\n \"metadata\": { \"source\": \"blog\", \"category\": \"technology\" }\n }\n ],\n \"embedding_model\": \"openai/text-embedding-3-small\"\n}\n```\n\n**Response:**\n\n```json\n{\n \"inserted\": 1,\n \"ids\": [\"550e8400-e29b-41d4-a716-446655440000\"]\n}\n```\n\n### `POST /collections/:collection/search`\n\nVector similarity search:\n\n```http\nPOST /api/v1/collections/my_docs/search\nContent-Type: application/json\nAuthorization: Bearer ndb_live_...\n\n{\n \"query\": \"AI-native database for semantic search\",\n \"limit\": 10,\n \"min_similarity\": 0.7,\n \"filters\": {\n \"category\": \"technology\"\n },\n \"embedding_model\": \"openai/text-embedding-3-small\",\n \"include_metadata\": true\n}\n```\n\n**Response:**\n\n```json\n{\n \"results\": [\n {\n \"id\": \"550e8400-e29b-41d4-a716-446655440000\",\n \"content\": \"NeuralDB is an AI-native database\",\n \"similarity\": 0.956,\n \"metadata\": { \"source\": \"blog\", \"category\": \"technology\" }\n }\n ],\n \"query_embedding_model\": \"openai/text-embedding-3-small\",\n \"latency_ms\": 2.8\n}\n```\n\n### `GET /collections/:collection/documents/:id`\n\nRetrieve a document by ID:\n\n```http\nGET /api/v1/collections/my_docs/documents/550e8400-e29b-41d4-a716-446655440000\nAuthorization: Bearer ndb_live_...\n```\n\n### `PUT /collections/:collection/documents/:id`\n\nUpdate a document:\n\n```http\nPUT /api/v1/collections/my_docs/documents/550e8400-...\nContent-Type: application/json\nAuthorization: Bearer ndb_live_...\n\n{\n \"content\": \"Updated content\",\n \"metadata\": { \"source\": \"blog\", \"updated\": true },\n \"regenerate_embedding\": true\n}\n```\n\n### `DELETE /collections/:collection/documents/:id`\n\nDelete a document:\n\n```http\nDELETE /api/v1/collections/my_docs/documents/550e8400-...\nAuthorization: Bearer ndb_live_...\n```\n\n## Batch Operations\n\n### `POST /collections/:collection/documents/batch`\n\nInsert up to 1,000 documents in a single request:\n\n```http\nPOST /api/v1/collections/my_docs/documents/batch\nContent-Type: application/json\n\n{\n \"documents\": [\n { \"content\": \"Document 1\", \"metadata\": {} },\n { \"content\": \"Document 2\", \"metadata\": {} }\n ]\n}\n```\n\n## Error Responses\n\nAll errors return a consistent JSON structure:\n\n```json\n{\n \"error\": {\n \"code\": \"QUERY_ERROR\",\n \"message\": \"relation \\\"nonexistent\\\" does not exist\",\n \"detail\": \"ERROR: relation \\\"nonexistent\\\" does not exist at character 15\",\n \"status\": 400\n }\n}\n```\n\n### Error Codes\n\n| HTTP Status | Error Code | Description |\n|-------------|-----------|-------------|\n| 400 | `QUERY_ERROR` | Invalid NQL query |\n| 400 | `VALIDATION_ERROR` | Request body validation failed |\n| 401 | `UNAUTHORIZED` | Missing or invalid API key |\n| 403 | `FORBIDDEN` | Insufficient role permissions |\n| 404 | `NOT_FOUND` | Document o"
|
|
}
|
|
] |