Data Engineering Portfolio

Introduction

This repository contains numerous work examples of code I use in my day to day work as a data engineer, all of which has been modified as minimum reproducible examples. My favourite tools are Snowflake, Python, and dbt, and I also have an interest in DevOps as it pertains to data engineering.

     

Table of Contents

• Python Examples
  • Snowpark example - backload data from SQL Server
  • Snowpark example - backload data from API
  • Automated SQL insert statements from a CSV file
  • Extract data from SQL Server, transform, and load to Snowflake
  • Batch load JSON files to Snowflake
  • SQL Server data Pull - 100 Records from every view in a database
• SQL Examples
  • Only grant permissions on tables with > 0 rows of data - Snowflake
  • Auto Ingest Snowpipe from Azure Blob to Snowflake
  • Shorten large union queries using Snowflake
  • Basic Snowflake CDC Pipeline using Streams and Tasks
  • Find missing dates in a date field - Snowflake
  • Snowflake data pipeline from internal stage
• Snowflake CI/CD using Azure Pipelines - SQLFluff testing, build and deploy using SnowSQL
• SQLFluff and yamllint pipelines for a dbt project

Usage

# Clone the repository
$ git clone https://github.com/MartyC-137/Data-Engineering.git

# Connect to the repository
$ cd Data-Engineering