DoraDB

DoraDB is an attempt to build a modern and fast storage engine in Rust from scratch. It is work in progress.

Goal

Build a modern and fast storage engine.

The storage engine is designed as a hybrid engine managing both in-memory row store and on-disk column store, with full transactional support across all data.

Quick Start

The snippets below assume an async function returning doradb_storage::Result<()>. DoraDB is runtime agnostic, so you can run these futures on whichever async runtime your application already uses. For a complete runnable version, see quick_start.rs or run cargo run --example quick_start.

Create a table with a schema and indexes, then drop it from an idle session.

use doradb_storage::{
    Engine, EngineConfig, StorageColumnFlags, StorageColumnSpec, StorageIndexFlags,
    StorageIndexKey, StorageIndexSpec, StorageTableSpec, ValKind,
};

let engine = Engine::bootstrap(
    EngineConfig::default().storage_root("target/doradb-quick-start"),
)
.await?;
let mut session = engine.new_session()?;

let created = session
    .create_table(
        StorageTableSpec::new(vec![
            StorageColumnSpec::new(ValKind::I32, StorageColumnFlags::empty()),
            StorageColumnSpec::new(ValKind::VarByte, StorageColumnFlags::empty()),
        ]),
        vec![
            StorageIndexSpec::new(vec![StorageIndexKey::new(0)], StorageIndexFlags::UK),
            StorageIndexSpec::new(vec![StorageIndexKey::new(1)], StorageIndexFlags::empty()),
        ],
    )
    .await?;
let table_id = created.table_id();
let id_index = created.index_ids()[0];
let name_index = created.index_ids()[1];

session.drop_table(table_id).await?;
session.close().await?;
engine.shutdown()?;

Insert, update, and delete rows through direct transaction methods.

use doradb_storage::{TableIndex, UpdateCol, Val};

let mut trx = session.begin_trx()?;

trx.table_insert_mvcc(table_id, vec![Val::from(1i32), Val::try_from("alice")?])
    .await?;

let key = [Val::from(1i32)];
let updated = trx
    .table_update_unique_mvcc(
        TableIndex(table_id, id_index),
        &key,
        vec![UpdateCol {
            idx: 1,
            val: Val::try_from("ada")?,
        }],
    )
    .await?;
assert!(updated.is_updated());

let deleted = trx
    .table_delete_unique_mvcc(TableIndex(table_id, id_index), &key)
    .await?;
assert!(deleted.is_deleted());

trx.commit().await?;

DML validation is enabled by default. Call trx.disable_dml_validation(true) only for input already proven against the table metadata; the setting applies to subsequent direct and streaming operations in that transaction. Call disable_dml_validation(false) to enable validation again.

Stream rows, read one unique-key row, and scan matching rows through a secondary index.

use doradb_storage::{ScanRowDecision, TableIndex, Val};

let mut trx = session.begin_trx()?;
let mut rows = Vec::new();

let mut stream = trx
    .table_scan_mvcc_stream(table_id, &[0, 1], |_| Ok(ScanRowDecision::Include))
    .await?;
while let Some(vals) = stream.next().await? {
    rows.push(vals);
}
drop(stream);

let id_key = [Val::from(1i32)];
let _row = trx
    .table_lookup_unique_mvcc(TableIndex(table_id, id_index), &id_key, &[0, 1])
    .await?;

let name_key = [Val::try_from("ada")?];
let _matching_rows = trx
    .table_index_lookup_mvcc(TableIndex(table_id, name_index), &name_key, &[0, 1])
    .await?
    .unwrap_rows();

trx.rollback().await?;

Design

• Public API
• Storage Architecture
• Transaction System
• Lock System
• Index Design
• Block Index
• Secondary Index
• Checkpoint
• Data Checkpoint
• Deletion Checkpoint
• Recovery
• Table File
• Buffer Pool
• Garbage Collect

Above documents describe the core design of this engine. Most parts are update-to-date. Some ideas are different from traditional database system. I'm glad to have discussions if someone is interested in details.

Code Structure

• buffer: Buffer pool implementation with async direct IO.
• catalog: Catalog of storage engine.
• compression: Compression algorithms for column store.
• file: Storage of table data, index and delete bitmap. The file is page based and organized as CoW B+Tree, to enable simple recovery and fast access.
• index: Block index and B+Tree index.
• io: Async direct IO system using Linux io_uring, which must be available in the runtime environment.
• latch: Async latch primitives including Mutex, RWLock and HybridLatch(enhanced RWLock with optimistic mode).
• lock: Metadata lock and table-level lock.
• log: Redo log record encoding, append, and initialization.
• lwc: Lightweight columnar format for on-disk warm data.
• recovery: Startup recovery coordination, redo stream replay, and recovered row state.
• row: In-memory row store and operations.
• table: Table of data, composite of block index, secondary index, buffer pool and table file. Support operations like index lookup, index scan, table scan, insert, delete, update, etc.
• trx: Transaction system, including transaction lifecycle, redo log integration, undo log, purge, and garbage collection.

Document-Driven AI Development Flow

Current development is heavily driven by document starting from Jan 2026, and implemneted by code agent. Every task assigned to agent has one associated task document located in docs/tasks. Larger features/refactors require RFC document inside docs/rfcs. Deferred tasks are put in docs/backlogs.

License

This project is licensed under either of

• Apache License, Version 2.0, (LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or https://opensource.org/licenses/MIT)

at your option.