NetbaDBPhase 73+
TutorialQueryArchitectureStorageRoadmap

Architecture

From Canonical Schema IR
to WAL-backed storage.

NetbaDB keeps application-language concerns at the frontend boundary. A Go, Rust, or future schema frontend produces the same Canonical Schema IR. The core does not inspect Go types or application Rust structs.

Application edge
Rust embedded SDK → Rust remote client → Go Protocol v2 client → netbadbd → NBOP v7
Compile
Parser → AST → HIR + type checking → Typed Relational IR
Plan and execute
Optimizer / Planner → IndexScan / HashJoin / NestedLoopJoin / IndexJoin → Synchronous executor
Transactions and storage
Transaction boundary → WAL + recovery → Heap MVCC / LSM / partitions / derived columnar

Architectural boundary

The following pipeline is the stable architectural surface:

Application language schema
        ↓
Language frontend / SDK
        ↓
Canonical Schema IR
        ↓
Parser → HIR + type checking
        ↓
Typed Relational IR
        ↓
Optimizer / Planner
        ↓
Executor
        ↓
Transaction boundary
        ↓
Storage

Current embedded path

The current path is synchronous. The query language is a deliberately small native subset and does not claim full SQL compatibility. The core does not depend on Go, a network runtime, JSON execution IR, or application-specific Rust structs.

Rust Schema API
    ↓
SELECT / JOIN / ORDER BY / GROUP BY + typed DML / Heap DDL
    ↓
Typed HIR
    ↓
Logical query / DML / DDL statement plan
    ↓
Scan + NestedLoopJoin / HashJoin / IndexJoin + sort + grouped aggregate
    ↓
Join / filter / sort / aggregate / projection / limit + mutation
    ↓
TableId → Heap / RANGE partition / LSM / derived Columnar
    ↓
Database coordinator + physical transaction lifecycle + versioned WAL
    ↓
optional DatabaseCommitSeq → published storage visibility vector
    ↓
Buffer pool (guards, pinning, dirty writeback)
    ↓
Slotted pages
    ↓
Page manager / database file

Canonical Schema IR

netbadb-schema stores database meaning in explicit Rust structs, independent of any application language. A column has a stable ColumnId, a name, a TypeSpec (physical type plus optional semantic name), nullability, and primary-key metadata.

Schema::new is the fallible construction path and delegates to Schema::validate. Validation rejects duplicate table/column IDs and names, empty names, and empty semantic-type names. Canonical names are frontend-independent UTF-8 identities; equality is exact and case-sensitive.

Each validated TableDef has canonical encoding version 1: it starts with NBTS, then an explicit little-endian version, table identity, and columns in declaration order. SHA-256 over those bytes is the 32-byte SchemaFingerprint. Rust enum discriminants, struct layout, Debug output, and map iteration order do not participate.

Compiler stages

The current query subset is compiled as follows:

source → AST → resolved / type-checked HIR → logical plan → physical plan

HIR owns source-level resolution and semantic type checking. Relational IR owns relational meaning and column provenance. The planner selects sequential or index scans, NestedLoopJoin, HashJoin, or Index Nested-Loop Join for analyzed Scan × Scan INNER JOIN. The executor evaluates typed expressions against rows from storage.

Layers pass IDs and owned values. They do not spread long-lived references to pages, frames, or tuples into the planner, executor, or catalog.

Dependency direction

In the graph, A → B means A depends on B. Lower layers must not depend on higher-level policy. In particular, storage must not depend on the planner or executor, and the executor must not depend on an SDK.

netbadb-types

Shared IDs, physical types, and semantic types

crates/netbadb-types

netbadb-schema

Language-independent Canonical Schema IR

crates/netbadb-schema

netbadb-parser

Small typed-query AST and parser

crates/netbadb-parser

netbadb-hir

Name resolution and nominal type checking

crates/netbadb-hir

netbadb-rel

Typed logical relational IR

crates/netbadb-rel

netbadb-compiler

AST → HIR → logical plan

crates/netbadb-compiler

netbadb-planner

Logical plan → physical plan, including IndexScan, HashJoin, IndexJoin, and ColumnarScan

crates/netbadb-planner

netbadb-schema-spec

SDK Schema Spec v1/v2 parsing and fingerprints

crates/netbadb-schema-spec

netbadb-tooling

Stable schema-driven SQL diagnostics

crates/netbadb-tooling

netbadb-inspect

Catalog and plan inspection DTOs, including Inspection JSON v7

crates/netbadb-inspect

netbadb-protocol

Protocol v2 binary wire contract; v1 is frozen and rejected

crates/netbadb-protocol

netbadb-client

Synchronous Protocol v2 remote client

crates/netbadb-client

netbadb-server

Sessions, authorization, blocking TCP, and Unix NBOP v7 operator plane

crates/netbadb-server

netbadb-codegen

Schema Spec v1/v2 validation and Go source generation

crates/netbadb-codegen

netbadb-index

Typed B+Tree ordering, nodes, codecs, and splits

crates/netbadb-index

netbadb-storage

Heap MVCC, LSM, derived columnar projections, WAL, pages, buffer pool, and persistent B+Tree

crates/netbadb-storage

netbadb-executor

Synchronous physical-plan execution

crates/netbadb-executor

netbadb-core

Native embedded Database API, schema catalog, coordinator, and columnar maintenance

crates/netbadb-core

netbadb-sdk

Embedded and remote application façade

sdk/rust

netbadbd

Standalone manifest-driven TCP server

cmd/netbadbd

netbadb

Offline local inspection CLI

cmd/netbadb

netbadb-lsp

Diagnostics-only stdio language server

cmd/netbadb-lsp

Cross-language strategy

Go is an application language, not an implementation language. The support boundary is:

Rust: native core, embedded SDK, and Protocol v2 remote client
Go: independent Protocol v2 client and generated typed bindings

sdk/go is an independent standard-library client. Generated bindings validate result order, names, physical and semantic types, and nullability. They do not generate SQL or query-builder APIs.

Design priorities

Correctness, explicit invariants, and type safety take precedence over convenience. Features are introduced as complete, testable vertical slices. Unimplemented components are not represented as finished APIs.