josephmachado/data_engineering_best_practices
josephmachado/data_engineering_best_practices has 228 GitHub stars and 49 forks. GitHot measured 0 stars in 24 hours, 0 in 7 days, and 0 in 15 days. Last checked 2026-11-22 05:50 UTC. Last push 2024-02-24.
Progression mesurée des étoiles
- in 24h
- 0
- in 7 days
- 0
- in 15 days
- 0
- in 1 month
- Historique en cours
Variations nettes sur des périodes qui se chevauchent, pas des totaux quotidiens. Les mesures absentes restent vides. Les étoiles indiquent l’intérêt, pas la qualité.
À propos
Sample project to demonstrate data engineering best practices
DonnéesREADME.md
OriginalData Engineering Best Practices
Code for blog at Data Engineering Best Practices - #1. Data flow & Code
Project
Assume we are extracting customer and order information from upstream sources and creating a daily report of the number of orders.

Setup
If you'd like to code along, you'll need
Prerequisite:
- git version >= 2.37.1
- Docker version >= 20.10.17 and Docker compose v2 version >= v2.10.2. Make sure that docker is running using
docker ps - pgcli
Run the following commands via the terminal. If you are using Windows, use WSL to set up Ubuntu and run the following commands via that terminal.
git clone https://github.com/josephmachado/data_engineering_best_practices.git
cd data_engineering_best_practices
make up # Spin up containers
make ddl # Create tables & views
make ci # Run checks & tests
make etl # Run etl
make spark-sh # Spark shell to check created tables
spark.sql("select partition from adventureworks.sales_mart group by 1").show() // should be the number of times you ran `make etl`
spark.sql("select count(*) from businessintelligence.sales_mart").show() // 59
spark.sql("select count(*) from adventureworks.dim_customer").show() // 1000 * num of etl runs
spark.sql("select count(*) from adventureworks.fct_orders").show() // 10000 * num of etl runs
:q // Quit scala shell
You can see the results of DQ checks, using make meta
select * from ge_validations_store limit 1;
exit
Use make down to spin down containers.
Architecture

Explorer des projets similaires
Chargement…