How To Become a Data Engineer

💼 If you want to land a data engineering role, visit getdataengineeringjobs.com. It lists only data engineering jobs, collected from company career pages and job boards, which you can filter by tech stack, salary, remote policy and visa sponsorship.

Useful articles

• The AI Hierarchy of Needs
• The Rise of Data Engineer
• The Downfall of the Data Engineer
• A Beginner’s Guide to Data Engineering
  • Part I
  • Part II
  • Part III
• Functional Data Engineering — a modern paradigm for batch data processing
• How to become a Data Engineer Ru, En
• Introduction to Apache Airflow Ru, En
• Apache Airflow Alternatives

Talks

• Data Engineering Principles - Build frameworks not pipelines by Gatis Seja
• Functional Data Engineering - A Set of Best Practices by Maxime Beauchemin
• Advanced Data Engineering Patterns with Apache Airflow by Maxime Beauchemin
• Creating a Data Engineering Culture by Jesse Anderson
• Streaming 101: Hello Streaming by Josh Fischer

Algorithms & Data Structures

• Algorithmic Toolbox in Russian
• Data Structures in Russian
• Data Structures & Algorithms Specialization on Coursera
• Algorithms Specialization from Stanford on Coursera

SQL

• Comprehensive SQL Tutorial by Mode Analytics
• SQL Practice on Leetcode
• Modern SQL a website about modern SQL syntax
• Introduction to Window Functions En, Ru

Programming

• Scala School by Twitter
• Fluent Python intermediate level book about Python
• Intro to Scala in Russian on Stepik by Tinkoff Bank
• The Hitchhiker’s Guide to Python by Kenneth Reitz & Tanya Schlusser
• Learn Python 3 The Hard Way by Zed A. Shaw

Databases

• Intro to Database Systems by Carnegie Mellon University
• Advanced Database Systems by Carnegie Mellon University
• On Disk IO
  • I. Flavors of IO
  • II. More Flavours of IO
  • III. LSM Trees
  • IV. B-Trees and RUM Conjecture
  • V. Access Patterns in LSM Trees

Distributed Systems

• Distributed systems for fun and profit by Mikito Takada
• Distributed Systems by Maarten van Steen & Andrew S. Tanenbaum
• CSE138: Distributed Systems by Lindsey Kuper
• CS 436: Distributed Computer Systems by University of Waterloo
• MIT 6.824: Distributed Systems by Robert Morris from MIT
• Distributed consensus reading list maintained by Heidi Howard from University of Cambridge

Books

• Design Data-Intensive Applications by Martin Kleppmann
• Fundamentals of Data Engineering: Plan and Build Robust Data Systems by Joe Reis & Matt Housley
• Introduction to Algorithms by Thomas Cormen
• The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling
• Star Schema The Complete Reference
• Database Internals: A Deep Dive into How Distributed Data Systems Work
• Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing
• A Philosophy of Software Design
• Grokking Streaming Systems by Josh Fischer & Ning Wang
• Guide to High Performance Distributed Computing by K.G. Srinivasa & Anil Kumar Muppalla
• Data Pipelines with Apache Airflow by Bas P. Harenslak and Julian Rutger de Ruiter

Courses

• Data Engineering on Google Cloud Platform Specialization by Google
• Data Engineer Nanodegree by Udacity
• Data Engineering with Python by DataCamp

Blogs

• Martin Kleppmann author of Designing Data-Intensive Application
• BaseDS by Vaidehi Joshi about Distributed Systems

Tools

• Apache Airflow is a platform to programmatically author, schedule and monitor workflows in Python
• Apache Spark is a unified analytics engine for large-scale data processing
• Apache Kafka is a distributed streaming platform
• Luigi is a Python package that helps you build complex pipelines of batch jobs.
• Dagster.io is a system for building modern data applications.
• Prefect includes everything you need to create and run data applications.
• Metaflow build and manage real-life data science projects with ease
• lakeFS build repeatable, atomic and versioned data lake operations – from complex ETL jobs to data science and analytics.

Cloud Platforms

• Amazon Web Services
• Google Cloud Platform
• Microsoft Azure
• Yandex Cloud
• DigitalOcean
• IBM Cloud

Communities

• data Engineering - telegram chat about data engineering
• Data Engineering Subreddit - subreddit about data engineering

Data Engineering Jobs

• Data Engineering jobs

Other

• Data Engineering Podcast

Newsletters & Digests

• DataEng Telegram channel - Telegram channel about data engineering (rus/eng)
• Data Engineering Weekly
• Data Elixir - Data Elixir is an email newsletter that keeps you on top of the tools and trends in Data Science.