cybergeekgyan/Data-Engineering-Portfolio
cybergeekgyan/Data-Engineering-Portfolio has 34 GitHub stars and 13 forks. GitHot measured 0 stars in 24 hours, 0 in 7 days, and 0 in 15 days. Last checked 2026-11-22 12:41 UTC. Last push 2024-03-25.
Gemessener Sternzuwachs
- in 24h
- 0
- in 7 days
- 0
- in 15 days
- 0
- in 1 month
- Verlauf wird gesammelt
Nettoänderungen in überlappenden Zeiträumen, keine Tagessummen. Fehlende Messwerte bleiben leer. Sterne zeigen Interesse, nicht Qualität.
Über uns
Data Engineering portfolio projects, resources used to study data tools...
DatenREADME.md
Original🔥Data-Engineering-Resources and Projects
Data Engineering Workflow
Big Data Architecture
📚Books
- Designing Data-Intensive Applications
- Fundamentals of Data Engineering
- The Data Warehouse Toolkit
- Cracking the Data Engineering Interview
- Data Engineering with Python
- Data Pipelines with Apache Airflow
- The Data Warehouse Toolkit
- Big Data: Principles and Best Practices of Scalable Real-Time Data Systems
- Designing Data-Intensive Applications
🧰Tools for Data Engineers
Basic Skills:
Linux,Git & GitHub,Computer Networking,Cloud Computing,Network & Security,Agile DevelopmentAdvanced Skills (Good to Know):
Data Lake & Data WareHouse Concepts,REST APIs,Databases(SQL & NoSQL)Programming Languages:
Python,SQL,Java,ScalaDatabases:
PostgreSQL,MongoDB,Neo4j,Redis,Cassandra,Apache HBase,Snowflake,InfluxDBData Ingestion:
Apache Kafka,Flume,Logstash,Airbyte,Apache Spark,Talend,InformaticaData Tranformation:
Python,Pandas,SQL,Apache Spark,Hive,dbt,Matillion,PigData Preprocessing:
Apache Spark,Apache Hadoop,Apache FlinkData Orchestration:
Apache Airflow,LuigiData Storage:
Data Lake: AWS S3, Azure Blob Storage, Google Cloud Storage,Data Warehouse: Snowflake, Google BigQuery, Amazon Redshift, Apache HiveData Visualization:
Tableau,PowerBI,LookerDataOps:
Docker,Kubernetes,Jenkins
Here's what's on the menu: 👇
- 🐍 Python,
- 📊 SQL,
- 🛠️ MySQL,
- 🌳 MongoDB,
- 🔥 PySpark,
- 🎈 Bash,
- 🌬️ Airflow,
- ☕ Apache Kafka,
- 🐙 Git,
- 🐈 GitHub,
- ⚙️ CICD basics,
- 🏬 Data Warehousing,
- 🛠️ DBT,
- 🌊 Data Lakes,
- 📘 DataBricks,
- ☁️ Azure Databricks,
- ❄️ Snowflake,
- 🌪️ Apache NiFi,
- 🌐 Debezium
Master Python: https://lnkd.in/d-pZPyf5
Learn SQL: https://lnkd.in/dzAiRF-x
Get hands-on with MySQL: https://lnkd.in/ddpSkUhc
Dive into MongoDB: https://lnkd.in/dHQ4VC2E
Master PySpark: https://lnkd.in/d7fgs7dE
Discover Bash, Airflow & Kafka: https://lnkd.in/dDhuEqQE
Master Git & GitHub: https://lnkd.in/dqJ7J3kN
Understand CICD basics: https://lnkd.in/dcfKBmCa
Decode Data Warehousing: https://lnkd.in/dPVRDJT5
Learn DBT: https://lnkd.in/eG9eaEuE
Understand Data Lakes: https://lnkd.in/dtZKJ4d6
Explore DataBricks: https://lnkd.in/dCBiQXPR
Learn Azure Databricks: https://lnkd.in/dzmwBs4Y
Master Snowflake: https://lnkd.in/dDBeddVy
Explore Apache NiFi: https://lnkd.in/de7bvnSt
📙Projects
🔶 Free Learning Resources
| Tools | Link | Used for | Official Docs | Youtube |
|---|---|---|---|---|
| DBMS | - MySQL - MongoDB | |||
| SQL | https://lnkd.in/dzAiRF-x | |||
| Python | https://lnkd.in/d-pZPyf5 | |||
| Linux | ||||
| Data Warehouse & Lake Concepts | - Data Warehouse - Data Lakes | |||
| Data Pipelines | ||||
| DBT | https://lnkd.in/eG9eaEuE | |||
| PySpark | https://lnkd.in/d7fgs7dE | |||
| Kafka | ||||
| Apache Nifi | https://lnkd.in/de7bvnSt | |||
| Airflow | ||||
| Databricks | https://lnkd.in/dCBiQXPR | |||
| Snowflake | https://lnkd.in/dDBeddVy | |||
| Cloud Computing Concepts | ||||
| Distributed Systems fundamentals | ||||
| AWS | ||||
| Azure | ||||
| GCP | ||||
| Git & GitHub | https://lnkd.in/dqJ7J3kN | |||
| CI/CD | https://lnkd.in/dcfKBmCa | |||
| Jenkins | ||||
| Github Actions | ||||
| Terraform | ||||
| Sonarqube | ||||
| Docker | ||||
| Kubernetes | ||||
| Power BI | ||||
| Tableau | ||||
| Apache Superset | ||||
| Prometheus | ||||
| Graphana | ||||
| Datadog |
💼 Read Real-World Case Studies -> Tech Blogs
- Netflix - https://netflixtechblog.medium.com/
- AWS - https://aws.amazon.com/solutions/case-studies/
- GCP - https://cloud.google.com/customers
- Azure - https://azure.microsoft.com/en-us/resources/customer-stories/
- Spotify - https://engineering.atspotify.com/category/data/
- MongoDB - https://www.mongodb.com/blog/all
- Swiggy - https://bytes.swiggy.com/the-swiggy-delivery-challenge-part-one-6a2abb4f82f6 - https://bytes.swiggy.com/swiggy-distance-service-9868dcf613f4 - https://bytes.swiggy.com/the-tech-that-brings-you-your-food-1a7926229886
- Zomato - https://blog.zomato.com/
Ähnliche Projekte entdecken
Wird geladen…