Data Engineer Prep

Prep for your next data engineering interview. Work through PySpark notebooks framed as real problems from Zephyr Coffee Co. (a fictional 200-store chain with messy data), review the theory docs before senior rounds, drill the quizzes the night before.

Built in the open. Contributions welcome — see below.

---

📚 Navigate

• pyspark/ — the PySpark module (start here)
• data_modeling/ — dimensional modeling, SCDs, star vs snowflake, grain
• ai_for_data_engineering/ — ⭐ the DE work behind LLMs: RAG/agents (using LLMs) + pre-training & SFT data (building LLMs)
• company_interviews/ — company-wise DE interview patterns (Ola, Flipkart, Swiggy, PhonePe, Jio)
• dsa/ — 🚀 authentic DE DSA syllabus & coding case studies (Google, stream windowing, reconciliation)
• ZEPHYR.md — the fictional company whose data runs through every notebook
• Roadmap — what's coming next
• Resources — thought leaders + resume examples
• Community — 💬 WhatsApp group for serious DEs
• Contributing

---

What's in here now

pyspark/

The PySpark module. READMEs inside guide you through it based on your level (beginner / intermediate / senior).

Hands-on notebooks (each framed as a Slack message from a Zephyr colleague asking you to solve a realistic problem):

• Syntax cheatsheet — 30-min flat reference
• Window functions — consecutive months, churn detection, top-N per group
• Joins — type mismatches, broadcast, skew detection, salting

Theory docs (10-min night-before-interview reviews):

• Shuffle & partitioning
• Memory management
• Catalyst & AQE
• Data skew playbook
• Spark UI & .explain() debugging

Self-check quizzes (collapsible Q&A, 🟢 basics → ⚡ senior judgment):

• Window functions · Joins · Memory · Catalyst/AQE · Skew · Spark UI

dsa/

The Data Engineering DSA module. Focuses strictly on patterns required to build scalable pipelines rather than general software engineering puzzles.

• Syllabus — The authentic DE DSA Syllabus mapping 6 essential patterns (Heaps, Sliding Window, Set Reconciliation, Sessionization, Two Pointers, Out-of-Core Processing) to LeetCode questions.
• Google Case Study — Deep dive into an intense 3-problem Google technical screen with production-ready Python solutions and distributed scaling strategies.

---

Roadmap

Phase 2 (next, no dates):

• Null handling & deduplication notebook (Zephyr's 2023 POS duplicate incident)
• Nested data notebook (exploding loyalty event structs)
• Structured streaming notebook
• Delta Lake notebook
• Quiz + theory coverage for each

Phase 3:

• SQL module (window functions in SQL, gaps-and-islands, SCDs, query optimization)
• Python for DE module (collections, generators, pandas↔Spark, testing)
• System design scenarios for DE interviews
• ✅ DE DSA Coding Bank & Syllabus (Google case studies, stream windowing, set reconciliation, out-of-core sharding)

The repo aims to be honest about what's built and what's not. No fake timelines.

---

Resources for Data Engineers

Thought leaders worth following

1. Sumit Mittal — Founder of BigDataBySumit
2. Joe Reis — Co-author of Fundamentals of Data Engineering
3. Zach Wilson — Data engineering specialist
4. Shashank Mishra — Data engineer & educator
5. Gowtham SB — Big data & cloud
6. Manish Kumar - For questions and interview experience
7. Darshil Parmar - For Crisp DE Videos
8. Ansh Lamba - Best for Azure and Databricks

Resource That I love

1. Data Pathshala Preparation of Data Engineering by Manish Kumar
2. Data Engineer Handbook DE Concepts

Resume examples

• Manish's DE resume — well-structured, shows skills/projects/experience clearly
• My resume
• Jake Overleaf - Open as template then edit it as your resume on latex

---

💬 Community

If you're serious about data engineering — interviewing, building, learning in public — I'm building a small WhatsApp community for it. Join here:

👉 WhatsApp DE community

Small and serious > big and noisy.

---

Contributing

Typo fixes, clearer explanations, new quiz questions, Zephyr scenario ideas, and blog-link additions (with a one-line justification for why it beats what's already linked) are all welcome. Open an issue or a PR.

Please don't send: random link dumps, self-promotional content, or AI-generated filler. The curation is the point — every external link in this repo was added because it's genuinely the best free resource for that topic, not because it exists.