Awesome Data Engineering Skills

A curated, portable collection of Agent Skills that make AI coding agents genuinely useful for data engineering work — dbt, Airflow, Dagster, Spark, Snowflake, BigQuery, Databricks, data quality, idempotency, backfills, and pipeline operations.

Each skill is a folder with a SKILL.md the agent loads only when relevant, so your context stays lean until you need the expertise. Skills follow the open Agent Skills standard and work across Claude Code, Cursor, Codex, GitHub Copilot, Gemini CLI, and other compatible agents.

Built from real, recurring data-engineering pain points (idempotency & backfills, silent schema changes, data quality incidents, warehouse cost, testing pipelines) — not generic advice.

Why skills for data engineering?

Data bugs are expensive and quiet: a non-idempotent retry duplicates a fact table, a backfill rewrites yesterday's revenue, a SELECT * becomes a recurring bill. These skills encode the correct default so the agent gets it right the first time.

Skill catalog

SQL & data modeling

Skill	Use it when
optimizing-sql-queries	A query is slow/expensive, scans too much, or spills — across Postgres, Snowflake, BigQuery, Spark, Redshift
modeling-dimensional-data	Designing star/snowflake schemas, choosing grain, or tracking history with SCDs
writing-idempotent-transformations	A retry could duplicate data, or a load must be backfill-safe

dbt

Skill	Use it when
building-dbt-models	Creating/refactoring models, choosing materialization, writing incremental logic
testing-dbt-projects	Adding tests, catching data quality regressions, source freshness
debugging-dbt-runs	dbt run/build fails, incremental is stale/duplicated, CI differs from local
documenting-dbt-models	Adding descriptions, exposures, and generated docs/lineage

Orchestration

Skill	Use it when
authoring-airflow-dags	Writing DAGs, scheduling, retries/backfills, fixing non-idempotent tasks
debugging-airflow-pipelines	A task fails or is stuck queued, scheduler won't run, zombie tasks
building-dagster-assets	Building software-defined assets, partitions, resources, asset checks
orchestrating-prefect-flows	Writing Prefect flows/tasks, retries, caching, deployments/schedules

Distributed processing

Skill	Use it when
optimizing-pyspark-jobs	A Spark job is slow, spills, OOMs, or has skew/large shuffles
engineering-databricks-pipelines	Building Delta/DLT pipelines, Auto Loader, Unity Catalog

Lakehouse & storage

Skill	Use it when
designing-medallion-architecture	Organizing a lakehouse into bronze/silver/gold layers
building-iceberg-tables	Creating/maintaining Iceberg tables, partitioning, compaction, time travel
optimizing-parquet-storage	Slow/costly Parquet reads, small-files problem, compression/layout

Warehouses & cost

Skill	Use it when
optimizing-snowflake-workloads	Snowflake is slow/expensive, warehouses spill/queue, sizing decisions
optimizing-bigquery-queries	BigQuery bytes billed are high or a query full-scans

Data quality & contracts

Skill	Use it when
implementing-data-quality-checks	Catching bad data before consumers — freshness, volume, schema, integrity
designing-data-contracts	A producer change could silently break downstream pipelines
handling-schema-evolution	Adding/renaming/retyping columns without breaking readers

Ingestion & streaming

Skill	Use it when
building-ingestion-pipelines	Extracting from DBs/APIs/files — incremental, CDC, watermarks, pagination
processing-streaming-data	Kafka/Spark/Flink streaming — delivery semantics, windowing, late data
building-kafka-consumers	Writing Kafka consumers/producers, offsets, consumer groups, dead-letter
implementing-cdc-with-debezium	Replicating an OLTP DB with CDC, capturing deletes, applying change streams

Pipeline engineering & ops

Skill	Use it when
debugging-data-pipelines	Numbers look wrong, data missing/duplicated, a dashboard is stale
designing-backfills-and-replays	Backfilling history or reprocessing after a fix — safely
implementing-pipeline-observability	Failures are found by stakeholders, not alerts; setting SLAs/monitoring
reviewing-data-pipeline-code	Reviewing a data PR for idempotency, grain, cost, tests, PII
implementing-data-cicd	Adding CI (lint/compile/test), dbt Slim CI, environment promotion
managing-data-lineage-openlineage	Cross-tool lineage, impact analysis, scoping incidents/backfills
migrating-legacy-etl	Migrating SSIS/Informatica/stored procs/on-prem to dbt/Spark/cloud
generating-synthetic-test-data	Realistic seeded test data, edge cases, referential integrity

Platform, governance & ML

Skill	Use it when
terraform-for-data-infra	Provisioning warehouses/buckets/IAM/orchestration as code
masking-pii-data	Masking/tokenizing PII, dynamic masking, GDPR/CCPA/HIPAA, deletion
building-feature-pipelines	ML features, point-in-time joins, offline/online parity, feature stores

Install

Skills live in this repo under skills/. Point your agent at them by copying or symlinking into the tool's skills directory. .agents/skills/ is recognized by the widest set of tools.

Quick install (macOS/Linux)

./scripts/install.sh claude     # -> ./.claude/skills/   (Claude Code)
./scripts/install.sh cursor     # -> ./.cursor/skills/   (Cursor)
./scripts/install.sh codex      # -> ./.codex/skills/    (Codex)
./scripts/install.sh copilot    # -> ./.github/skills/   (GitHub Copilot)
./scripts/install.sh agents     # -> ./.agents/skills/   (broadest support)
# add --user for a global install, --copy to copy instead of symlink

Per-tool skills directories

Tool	Project path	Personal (global) path
Claude Code	.claude/skills/ or .agents/skills/	~/.claude/skills/
Cursor	.cursor/skills/ or .agents/skills/ (also reads .claude/, .codex/)	~/.cursor/skills/, ~/.agents/skills/
Codex	.codex/skills/	~/.codex/skills/
GitHub Copilot	.github/skills/, .claude/skills/, or .agents/skills/	~/.copilot/skills/, ~/.agents/skills/

Claude Code plugin (optional)

Install the whole set as a plugin marketplace:

/plugin marketplace add <your-org>/awesome-data-engineering-skills
/plugin install data-engineering-skills@awesome-data-engineering-skills

How skills load

1. Discovery — the agent preloads each skill's name + description (~tiny).
2. Activation — when your task matches, it reads the full SKILL.md.
3. Resources — deeper references/*.md load only when needed.

You can also invoke a skill directly by typing / in chat and picking it by name.

Portability

Phase-1 skills use only the portable spec frontmatter (name, description) so they behave identically across tools. Tool-specific extensions (dynamic command injection, forked subagents, paths) are intentionally avoided. See CONTRIBUTING.md.

Contributing

New skills and improvements welcome — see CONTRIBUTING.md. Validate before opening a PR:

python scripts/validate_skills.py   # frontmatter, naming, references
python scripts/check_evals.py       # every skill has trigger/non-trigger evals

Each skill has description-tuning eval cases in evals/triggering.json (prompts that should and should not activate it) to keep discovery accurate.

Roadmap (backlog)

The initial catalog above covers the core plus the first expansion set. Candidate future skills: writing-great-expectations-suites · tuning-warehouse-costs · building-realtime-analytics · managing-data-catalogs · orchestrating-dbt-airflow.

License

Apache-2.0