josephmachado/data_engineering_best_practices
josephmachado/data_engineering_best_practices has 228 GitHub stars and 49 forks. GitHot measured 0 stars in 24 hours, 0 in 7 days, and 0 in 15 days. Last checked 2026-11-15 14:09 UTC. Last push 2024-02-24.
Measured star growth
- in 24h
- 0
- in 7 days
- 0
- in 15 days
- 0
- in 1 month
- Update delayed
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
Sample project to demonstrate data engineering best practices
DataREADME.md
OriginalData Engineering Best Practices
Code for blog at Data Engineering Best Practices - #1. Data flow & Code
Project
Assume we are extracting customer and order information from upstream sources and creating a daily report of the number of orders.

Setup
If you'd like to code along, you'll need
Prerequisite:
- git version >= 2.37.1
- Docker version >= 20.10.17 and Docker compose v2 version >= v2.10.2. Make sure that docker is running using
docker ps - pgcli
Run the following commands via the terminal. If you are using Windows, use WSL to set up Ubuntu and run the following commands via that terminal.
git clone https://github.com/josephmachado/data_engineering_best_practices.git
cd data_engineering_best_practices
make up # Spin up containers
make ddl # Create tables & views
make ci # Run checks & tests
make etl # Run etl
make spark-sh # Spark shell to check created tables
spark.sql("select partition from adventureworks.sales_mart group by 1").show() // should be the number of times you ran `make etl`
spark.sql("select count(*) from businessintelligence.sales_mart").show() // 59
spark.sql("select count(*) from adventureworks.dim_customer").show() // 1000 * num of etl runs
spark.sql("select count(*) from adventureworks.fct_orders").show() // 10000 * num of etl runs
:q // Quit scala shell
You can see the results of DQ checks, using make meta
select * from ge_validations_store limit 1;
exit
Use make down to spin down containers.
Architecture
