Apache Iceberg Table Provider for Apache DataFusion

This repository connects Apache Iceberg tables to Apache DataFusion. The iceberg-datafusion crate provides table providers for querying Iceberg data and for inserting rows through an Iceberg catalog.

Workspace

• crates/datafusion: the DataFusion integration library.
• crates/sqllogictest: SQL logic tests for the integration.
• crates/playground: a command-line SQL playground backed by DataFusion and Iceberg catalogs.

Query an existing table

Register IcebergTableProviderFactory in a DataFusion session, then point an external table at an existing Iceberg metadata file:

use std::sync::Arc;

use datafusion::execution::session_state::SessionStateBuilder;
use datafusion::prelude::SessionContext;
use iceberg_datafusion::IcebergTableProviderFactory;

async fn query_table() -> datafusion::error::Result<()> {
    let mut state = SessionStateBuilder::new().with_default_features().build();
    state.table_factories_mut().insert(
        "ICEBERG".to_string(),
        Arc::new(IcebergTableProviderFactory::new()),
    );
    let ctx = SessionContext::new_with_state(state);

    ctx.sql(
        "CREATE EXTERNAL TABLE trips STORED AS ICEBERG \
         LOCATION '/absolute/path/to/table/metadata/v1.metadata.json'",
    )
    .await?
    .collect()
    .await?;

    let batches = ctx
        .sql("SELECT * FROM trips LIMIT 10")
        .await?
        .collect()
        .await?;
    println!("{batches:?}");
    Ok(())
}

Replace the metadata path with one from an existing table whose data files are accessible to the process. External-table registration reads an existing table; it does not create one. For catalog-backed access and inserts, register an IcebergCatalogProvider with a configured Iceberg Catalog. See the integration tests for examples.

The workspace uses a pinned iceberg-rust Git revision. Applications that also depend on Iceberg crates should use the same revision shown in Cargo.toml so Cargo uses one Iceberg crate source.

Development

The repository's rust-toolchain.toml selects the Rust toolchain used by CI. From the repository root, run:

cargo fmt --all -- --check
cargo clippy --workspace --locked --all-targets -- -D warnings
cargo test --workspace --locked

This project is licensed under the Apache License, Version 2.0. See LICENSE and NOTICE.