Victor-Kipruto-Rop/SQL-for-Data
Victor-Kipruto-Rop/SQL-for-Data has 43 GitHub stars and 1 forks. GitHot measured 0 stars in 24 hours, 0 in 7 days, and 0 in 15 days. Last checked 2026-11-20 20:19 UTC. Last push 2026-04-02.
Gemessener Sternzuwachs
- in 24h
- 0
- in 7 days
- 0
- in 15 days
- 0
- in 1 month
- Verlauf wird gesammelt
Nettoänderungen in überlappenden Zeiträumen, keine Tagessummen. Fehlende Messwerte bleiben leer. Sterne zeigen Interesse, nicht Qualität.
Über uns
SQL practice repository covering data analysis, joins, aggregations, window functions, and real-world query solving for data engineering and analytics.
DatenREADME.md
OriginalSQL Data Engineering Project
End-to-end SQL project for ingesting raw CSV data, transforming it into an OLTP model, loading a dimensional warehouse, and running analytics, quality checks, performance tests, and monitoring.
Project Layout
sql-data-engineering-project/
├── database/
├── data/
├── etl/
├── warehouse/
├── analytics/
├── data_quality/
├── performance/
├── monitoring/
├── tests/
└── docs/
Tech Stack
- PostgreSQL 14+
- SQL (psql-compatible scripts)
- Optional Python tools (linting/automation)
CI
- GitHub Actions workflow:
.github/workflows/ci.yml - Runs the full pipeline (
./run_all.sh) on pushes and pull requests tomain. - Scheduled monitoring workflow:
.github/workflows/monitoring_schedule.yml - Runs weekly and on manual trigger; publishes monitoring logs as workflow artifacts.
Quick Start
One command end-to-end (recommended):
chmod +x run_all.sh
./run_all.sh
Run with Docker (PostgreSQL + pipeline):
docker compose up -d postgres
docker compose --profile run run --rm pipeline
Manual execution:
- Create database (example):
createdb sql_data_engineering
Set environment values in
.env.Initialize core schemas and tables:
psql "$DATABASE_URL" -f database/schema.sql
psql "$DATABASE_URL" -f database/tables.sql
psql "$DATABASE_URL" -f database/constraints.sql
psql "$DATABASE_URL" -f database/indexes.sql
- Load source data and run ETL:
psql "$DATABASE_URL" -f etl/extract.sql
psql "$DATABASE_URL" -f etl/transform.sql
psql "$DATABASE_URL" -f etl/load.sql
- Build warehouse model (includes SCD Type 2 update + fact load):
psql "$DATABASE_URL" -f warehouse/star_schema.sql
- Run quality checks, analytics, and tests:
psql "$DATABASE_URL" -f data_quality/validation_queries.sql
psql "$DATABASE_URL" -f analytics/revenue_analysis.sql
psql "$DATABASE_URL" -f tests/test_data_load.sql
psql "$DATABASE_URL" -f tests/test_scd_logic.sql
psql "$DATABASE_URL" -f tests/test_quality_checks.sql
psql "$DATABASE_URL" -f tests/test_customer_segmentation.sql
Pipeline Flow
- Extract CSVs into staging raw tables.
- Transform and standardize datatypes + deduplicate records.
- Load cleaned data into OLTP tables with upserts.
- Warehouse load dimensions/facts and apply SCD Type 2 for customers.
- Analyze KPIs and run fraud/retention/segmentation logic.
- Monitor row counts and anomaly signals.
Notes
- SQL is written for PostgreSQL.
etl/extract.sqluses\copy, so run withpsqlfrom project root.- Example data is included in
data/raw/. - Architecture and data model diagrams are generated from DOT sources:
docs/architecture_diagram.dotdocs/data_model.dot- Regenerate with
./docs/generate_diagrams.sh