Databricks Lakeflow Framework

Documentation | Sample Data Bundles

Project Description

The Lakeflow Framework is a metadata-driven framework for building Databricks Lakeflow Spark Declarative Pipelines. It uses a configuration-driven, pattern-based approach to support both batch and streaming workloads across the medallion architecture.

The framework supports centralized and domain-oriented operating models, and accommodates multiple modelling paradigms (including dimensional, Data Vault, and enterprise canonical models). It is designed for simplicity, performance, maintainability, and extensibility as the Databricks product evolves.

https://github.com/user-attachments/assets/eb95bf4c-ebb1-4064-afc7-3450facd7ec9

Why use Lakeflow Framework

• Configuration-driven pattern based pipeline delivery with reusable implementation patterns
• Support for batch and streaming pipelines across Bronze/Silver/Gold, aligned to your chosen modelling pattern
• Flexible for centralized and domain-oriented operating models

Prerequisites

• Access to a Databricks workspace
• Databricks CLI installed and authenticated (databricks auth login for your workspace, or a configured CLI profile)
• Familiarity with Databricks Lakeflow Spark Declarative Pipelines concepts

Quick start

Deploy the framework:

git clone https://github.com/databricks-solutions/lakeflow_framework.git
cd lakeflow_framework
databricks bundle deploy -t dev

Deploy samples (requires the framework above): see samples/README.md for bundle descriptions and deploy scripts (deploy.sh, deploy_feature_samples.sh, deploy_tpch.sh, and others).

cd samples
./deploy.sh

Full deployment steps, configuration options, and walkthroughs are in the public docs Getting Started section.

Repository structure

• docs/ — Sphinx documentation and versioned docs build tooling
• samples/ — example framework and pipeline bundles
• src/ — framework bundle root deployed to the workspace (framework.sourcePath in DAB)
  • lakeflow_framework/ — canonical Python package: runtime code, bundled default config (config/default/), and JSON schemas (schemas/). Prefer from lakeflow_framework... imports in new code.
  • *.py at src/ root — backward-compatibility shims for legacy bare imports (e.g. from constants import ...); removed at v1.0.0
  • local/ — customer-owned sparse config and extensions; never overwritten by upstream upgrades (src/local/README.md)

See Import conventions

Version compatibility

This project tracks Databricks Lakeflow Spark Declarative Pipelines capabilities and evolves with platform changes. Validate runtime, feature, and API compatibility against your target Databricks workspace and the latest project documentation before production rollout.

Project status and support

The framework is actively maintained. Databricks support does not cover this repository; issue support is best effort through GitHub issues.

Releases and changelog

• Releases: https://github.com/databricks-solutions/lakeflow_framework/releases
• Tags: https://github.com/databricks-solutions/lakeflow_framework/tags

Documentation

Please refer to the documentation for further details and an explanation of the samples. The documentation needs to be deployed as HTML or Markdown within your org before it can be used.

Local development quick start (contributors)

Clone the repository, then set up a Python 3.12 environment. This is a minimal local quick start — full contributor guidance (VS Code setup, lockfiles, deployment, and PR workflow) is in the documentation under Framework Development & Contributors.

git clone https://github.com/databricks-solutions/lakeflow_framework.git
cd lakeflow_framework
python -m venv .venv
source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install --require-hashes --no-deps -r requirements-dev.lock

For editable installs and IDE auto-complete: pip install -e ".[contrib]" (see Development Environment Setup).

Run unit tests:

pytest tests/ -m "not integration and not spark"

See also tests/README.md and docs/source/contributor_imports.rst in the repository.

Local docs development (optional)

Requires dev dependencies from requirements-dev.lock:

make -C docs html

How to get help

Databricks support doesn't cover this content. For questions or bugs, please open a GitHub issue and the team will help on a best effort basis.

License

© 2025 Databricks, Inc. All rights reserved. The source in this notebook is provided subject to the Databricks License [https://databricks.com/db-license-source]. All included or referenced third party libraries are subject to the licenses set forth below.