Skip to content
Projects

darshilparmar/python-for-data-engineering

View on GitHub

darshilparmar/python-for-data-engineering has 387 GitHub stars and 839 forks. GitHot measured -1 stars in 24 hours. Last checked 2026-11-20 10:51 UTC. Last push 2024-04-24.

How these counts are measured

Measured star growth

in 24h
-1
in 7 days
Collecting history
in 15 days
Collecting history
in 1 month
Collecting history

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

This repo contains all the code used in the Python for Data Engineering Course

Data

README.md

Original

Python for Data Engineering

This repo contains all the code used in the Python for Data Engineering Course

Code is free to use, if you want in-depth understanding of concepts then check out my python course designed to teach you how to think programmatically and perform data engineering tasks using python

Module 1: iPhone Data Analysis Project

Get right into it, don't waste time learning basics, and execute the complete project to boost your confidence.

Work on Python Data Analysis project on iPhone Apple Dataset, do some basic transformation and analysis.

Module 2: Python Basics

Build strong programming fundamentals by learning the basics of python

  • Working with Strings
  • Working with Numbers
  • Understanding DataTypes
  • Python Data Structures (List, Dictionary, Sets, Tuples)
  • Working with Loops (For & While)
  • Conditional Statements (IF-ELSE)

Module 3: Python Intermediate

Let's learn some intermediate concepts to build a strong foundation.

  • List Compherention
  • Understand Python Packages
  • Exception Handling
  • Working with a text file
  • Working with Functions
  • Lambda functions basics

Module 4: Python Advance

Let's learn some advanced topics and packages used by data engineer on a daily basis

  • Py Package - Numpy
  • Py Package - Pandas (Basics and Understanding Transformation with Project)

Module 5: Python for Data Engineering

Python for data engineering, learn how to work with different file formats and handle DateTime columns

  • Working with DateTime Formats
  • Working with different file formats (JSON, CSV, Excel, AVRO)
  • Scrape Amazon Data Using Python

Module 6: Spotfiy Data Pipeline End-To-End Python Data Engineering Project

Implement Complete Data Pipeline Data Engineering Project using Spotify

  • Integrating with Spotify API and extracting Data
  • Deploying code on AWS Lambda for Data Extraction
  • Adding trigger to run the extraction automatically
  • Writing transformation function
  • Building automated trigger on transformation function
  • Store files on S3 properly
  • Building Analytics Tables on data files using Glue and Athena