Tonbo

Website | Rust Doc | Blog | Community

Tonbo is an embedded database for serverless and edge runtimes. Your data is stored as Parquet on S3, coordination happens through a manifest, and compute stays fully stateless.

Why Tonbo?

Serverless compute is stateless, but your data isn't. Tonbo bridges this gap:

• Async-first: The entire storage and query engine is fully async, built for serverless and edge environments.
• No server to manage: Data lives on S3, coordination via manifest, compute is stateless
• Arrow-native: Define rich data type, declarative schemas, query with zero-copy RecordBatch
• Runs anywhere: Tokio, WASM, edge runtimes, or as a storage engine for building your own data infrastructure.
• Open formats: Standard Parquet files readable by any tool

When to use Tonbo?

• Build serverless or edge applications that need a durable state layer without running a database.
• Store append-heavy or event-like data directly in S3 and query it with low overhead.
• Embed a lightweight MVCC + Parquet storage engine inside your own data infrastructure.
• Run workloads in WASM or Cloudflare Workers that require structured persistence.

Quick Start

use tonbo::{db::{AwsCreds, ObjectSpec, S3Spec}, prelude::*};

#[derive(Record)]
struct User {
    #[metadata(k = "tonbo.key", v = "true")]
    id: String,
    name: String,
    score: Option<i64>,
}

// Open on S3
let s3 = S3Spec::new("my-bucket", "data/users", AwsCreds::from_env()?);
let db = DbBuilder::from_schema(User::schema())?
    .object_store(ObjectSpec::s3(s3))?.open().await?;

// Insert
let users = vec![User { id: "u1".into(), name: "Alice".into(), score: Some(100) }];
let mut builders = User::new_builders(users.len());
builders.append_rows(users);
db.ingest(builders.finish().into_record_batch()).await?;

// Query
let filter = Expr::gt("score", ScalarValue::from(80_i64));
let results = db.scan().filter(filter).collect().await?;

For local development, use .on_disk("/tmp/users")? instead. See examples/ for more.

Installation

cargo add [email protected] tokio

Or add to Cargo.toml:

[dependencies]
tonbo = "0.4.0-a1"
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }

Examples

Run with cargo run --example <name>:

• 01_basic: Define schema, insert, and query in 30 lines
• 02_transaction: MVCC transactions with upsert, delete, and read-your-writes
• 02b_snapshot: Consistent point-in-time reads while writes continue
• 03_filter: Predicates: eq, gt, in, is_null, and, or, not
• 04_s3: Store Parquet files on S3/R2/MinIO with zero server config
• 05_scan_options: Projection pushdown reads only the columns you need
• 06_composite_key: Multi-column keys for time-series and partitioned data
• 07_streaming: Process millions of rows without loading into memory
• 08_nested_types: Deep struct nesting + Lists stored as Arrow StructArray
• 09_time_travel: Query historical snapshots via MVCC timestamps
• 10_dynamic: Dynamic schemas without #[derive(Record)] (basic, metadata, composite, transactions)
• 11_observability: Tracing spans and structured logging with OpenTelemetry

Architecture

Tonbo implements a merge-tree optimized for object storage: writes go to WAL → MemTable → Parquet SSTables, with MVCC for snapshot isolation and a manifest for coordination via compare-and-swap:

• Stateless compute: A worker only needs to read and update the manifest; no long-lived coordinator is required.
• Object storage CAS: The manifest is committed using compare-and-swap on S3, so any function can safely participate in commits.
• Immutable data: Data files are write-once Parquet SSTables, which matches the strengths of S3 and other object stores.

See docs/overview.md for the full design.

Observability

Tonbo uses the tracing crate for structured logging and distributed tracing. As a library, Tonbo emits spans and events but never initializes a global subscriber - your application controls the observability configuration.

Quick setup for development:

tracing_subscriber::fmt()
    .with_env_filter("info,tonbo=debug")
    .init();

Production with JSON output:

tracing_subscriber::fmt()
    .json()
    .with_env_filter("info,tonbo=info")
    .init();

OpenTelemetry integration:

use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

let tracer = opentelemetry_otlp::new_pipeline()
    .tracing()
    .install_batch(opentelemetry_sdk::runtime::Tokio)?;

tracing_subscriber::registry()
    .with(tracing_opentelemetry::layer().with_tracer(tracer))
    .with(tracing_subscriber::fmt::layer())
    .init();

See examples/11_observability.rs for complete examples and docs/rfcs/0012-logs-traces.md for design details.

Documentation

• User Guide: tonbo.io/docs: tutorials and concepts
• API Reference: docs.rs/tonbo: full Rust API documentation
• RFCs: docs/rfcs/: design documents for contributors

Development

Coverage

Install coverage tooling once:

rustup component add llvm-tools-preview
cargo install cargo-llvm-cov --version 0.6.12 --locked

Run coverage locally:

cargo llvm-cov --workspace --lcov --output-path lcov.info --summary

Generate an HTML report:

cargo llvm-cov --workspace --html

Compaction Benchmark (Local FS)

Default run:

cargo bench --bench compaction_local

Example tuned run:

TONBO_COMPACTION_BENCH_INGEST_BATCHES=768 \
TONBO_COMPACTION_BENCH_ROWS_PER_BATCH=96 \
TONBO_COMPACTION_BENCH_KEY_SPACE=4096 \
TONBO_COMPACTION_BENCH_ARTIFACT_ITERATIONS=64 \
TONBO_COMPACTION_BENCH_CRITERION_SAMPLE_SIZE=30 \
TONBO_COMPACTION_BENCH_WAL_SYNC=always \
cargo bench --bench compaction_local

Benchmark configuration is env-only via TONBO_COMPACTION_BENCH_*. For benchmark setup, scenario wiring, and JSON artifact schema/output, see:

• benches/compaction_local.rs
• benches/compaction/common.rs
• docs/benchmark_results.md for the consolidated PR-facing summary
• benches/compaction/results/compaction_local_baseline.md for raw baseline evidence

Project status

Tonbo is currently in alpha. APIs may change, and we're actively iterating based on feedback. We recommend starting with development and non-critical workloads before moving to production.

Features

Storage

• Parquet files on object storage (S3, R2) or local filesystem
• Manifest-driven coordination (CAS commits, no server needed)
• (in-progress) Remote compaction (offload to serverless functions)
• (in-progress) Branching (git-like fork and merge for datasets)
• Time-window compaction strategy

Schema & Query

• Arrow-native schemas (#[derive(Record)] or dynamic Schema)
• Projection pushdown (read only needed columns)
• Zero-copy reads via Arrow RecordBatch
• (in-progress) Filter pushdown (predicates evaluated at storage layer)

Backends

• Local filesystem
• S3 / S3-compatible (R2, MinIO)
• (in-progress) OPFS (browser storage)

Runtime

• Async-first (Tokio)
• Edge runtimes (Deno, Cloudflare Workers)
• WebAssembly
• (in-progress) io_uring
• (in-progress) Python async bindings
• JavaScript/TypeScript async bindings

Integrations

• DataFusion TableProvider
• Postgres Foreign Data Wrapper

Observability

• Structured logging via tracing crate
• OpenTelemetry compatible (via tracing-opentelemetry)
• (in-progress) Async-aware spans for storage operations

License

Apache License 2.0. See LICENSE for details.