Skip to content
プロジェクト

imsanjoykb/ETL-Project

GitHubで見る

imsanjoykb/ETL-Project has 34 GitHub stars and 18 forks. Star changes are not measured yet. Last checked 2026-11-14 11:46 UTC. Last push 2021-09-07.

この数値の測り方

34 スター18 フォーク

計測したスター増加

24時間
履歴を収集中
7日間
履歴を収集中
15日間
履歴を収集中
1か月
履歴を収集中

期間ごとの純増数です。期間は重複しており、日別の値ではありません。未計測の値は空欄です。スター数は注目度の指標であり、品質の保証ではありません。

概要

The goal of this project is to illustrate Extract Transform Load (ETL) using Python and SQL. ETL is a process commonly done in computing, which takes raw data, cleans it and stores it for later use. The extraction phase targets and retrieves the data. Transform manipulates and cleans the data. Then load stores the data, typically in a data warehouse.

データ

README.md

オリジナル

Covid-19 Infection ETL Project

By Sanjoy Biswas

Project Proposal

Based upon the data compiled by John Hopkins University, I want to explore ''' Insert reasons here''' This will be done by extracting the CSV data and migrating it to a PostgreSQL Database.

Project Description

I found data from data.data.org that had been compiled from John Hopkins University. I filtered the data for March 2020 and evaluated the number of cases with respect to the deaths, recovery, and the confirmed cases.

Finding Data

All of the data that we used were from https://data.humdata.org/dataset/novel-coronavirus-2019-ncov-cases where they compiled by the John Hopkins University Center for Systems Science and Engineering (JHU CSSE) from various sources that include the World Health Organisations, Hong Kong Department of Health, European Centre for Disease Prevention and Control, etc. This data is always being updated so we are narrowing the scope to the month of March.

Data Cleanup and Analysis

  • TRANSFORMATION STEPS My transformation steps I needed to clean the data to be readable, presentable, and easy for me to query in the later stages. This was done by:

    • Developing a cleaning function in python that would select the data in the month of March 2020. This was applied to all datasets that I have.
    • All of the dates that have in the data sets we treated as values through the pd.melt function in Pandas.
    • Found a way of finding the daily increase with respect to each table. This value was converted from a float to an integer

  • LOADING STEPS

    • I established a connection to a local PostgreSQL server in our desktop to store the data
    • I have a schema that just makes the tables and we can confirm it throught engine.table_names()
    • I pushed the Pandas DataFrame to the local PostgreSQL server so I can retrieve and query the data in our Jupyter Notebook

  • Analysis / SQL Queries In this part, I want to find:

    • Top 5 countries with the most/least confirmed cases
    • Top 5 countries with the most/least deaths
    • Top 5 countries with the most/least recovered
    • Date in March with the most confirmed/deaths/recovered

関連プロジェクトを探す

Asabeneh/30-Days-Of-Python

30 Days of Python プログラミングチャレンジは、Python プログラミング言語を 30 日で学ぶためのステップバイステップのガイドです。このチャレンジは 100 日以上かかることもあります。自分のペースで進めてください。こちらの動画も役立つかもしれません: https://www.youtube.com/channel/UC7PNRuno1rzYPb1xLa4yktw

78,729 スター

grafana/grafana

オープンで組み合わせ可能なオブザーバビリティおよびデータ可視化プラットフォームです。Prometheus、Loki、Elasticsearch、InfluxDB、Postgres など、複数のソースからメトリクス、ログ、トレースを可視化できます。

77,898 スター

redis/redis

リアルタイムのデータ駆動型アプリケーションを構築する開発者にとって、Redis は最も選ばれ、最速で、機能が最も豊富なキャッシュ、データ構造サーバー、ドキュメントおよびベクトルクエリエンジンです。

77,491 スター

apache/superset

Apache Superset は、データ可視化とデータ探索のためのプラットフォームです。

75,523 スター