sonhmai/data-systems-design

= Data Systems Design and Implementation :idprefix: :idseparator: - :sectanchors: :sectlinks: :sectnumlevels: 6 :sectnums: :toc: macro :toclevels: 6 :toc-title:

Collection of useful notes, readings, videos, codes related to data systems design

toc::[]

== System Design Resources

=== Learning Resources Repos

• https://github.com/karanpratapsingh/system-design
• https://github.com/donnemartin/system-design-primer

=== Books

• https://www.amazon.com/Understanding-Distributed-Systems-Second-applications/dp/1838430210[Understanding Distributed Systems: What every developer should know about large distributed applications]
• https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321[Design Data-Intensive Applications]
• https://www.amazon.com/Patterns-Distributed-Systems-Addison-Wesley-Signature/dp/0138221987[Patterns of Distributed Systems. Unmesh Joshi]

=== Blogs

• https://eng.lyft.com/building-a-control-plane-for-lyfts-shared-development-environment-6a40266fcf5e[Building a Control Plane for Lyft’s Shared Development Environment]
• https://dgryski.medium.com/consistent-hashing-algorithmic-tradeoffs-ef6b8e2fcae8[Consistent Hashing: Algorithmic Tradeoffs]
• https://discord.com/blog/why-discord-is-switching-from-go-to-rust[Why Discord is switching from Go to Rust], interesting discussions on https://www.reddit.com/r/golang/comments/eywx4q/why_discord_is_switching_from_go_to_rust/[ Reddit thread], https://news.ycombinator.com/item?id=26227339[Hackernews].
• https://encore.dev/blog/queueing[Queueing: an interactive study of queueing strategies]
• https://muratbuffalo.blogspot.com/2024/07/understanding-performance-implications.html[Understanding the Performance Implications of Storage-Disaggregated Databases]
• https://www.somethingsimilar.com/2013/01/14/notes-on-distributed-systems-for-young-bloods/[Notes on Distributed Systems for Young Bloods]
• https://arnon.me/[Arnon blog]

Replication

• https://transactional.blog/blog/2024-data-replication-design-spectrum[Blog - Data Replication Design Spectrum]

Performance

• https://tigerbeetle.com/blog/a-friendly-abstraction-over-iouring-and-kqueue[A Programmer-Friendly I/O Abstraction Over io_uring and kqueue]

CAP

• https://www.youtube.com/watch?v=srOgpXECblk[Google I/O 2009 - Transactions Across Datacenters]: distributed systems, transactions, fault-tolerance, consistency, replication, sharding, two phase commit, consensus protocol with App Engine datastore use case.

=== Video Courses

• https://www.youtube.com/playlist?list=PLeKd45zvjcDFUEv_ohr_HdUFe97RItdiB[Distributed Systems Lecture Series by Martin Kleppmann]
• https://www.youtube.com/watch?v=cQP8WApzIQQ&list=PLrw6a1wE39_tb2fErI4-WkMbsvGQk9_UB[MIT 6.824: Distributed Systems Course, 2020.]

== Data Architecture

Blogs

• Grab, Enabling near real-time data analytics on the data lake (https://engineering.grab.com/enabling-near-realtime-data-analytics)
• https://www.decodable.co/blog/taxonomy-of-data-change-events[A Taxonomy Of Data Change Events]
• https://materialize.com/blog/operational-data-store/[What Happened to the Operational Data Store?]

Videos

• https://www.youtube.com/watch?v=h_pXIn3Jdak[Youtube, Real Time with Redpanda, Episode 4: Write path optimizations for tiered storage]

== Data Storage Terms

• Table Formats (Iceberg, Hudi, Delta, OneTable, etc.).
• Features (ACID transaction, Schema Evolution, Upsert, Time Travel, etc.).
• Tech ( Copy On Write vs Merge On Read, Compaction, Vacuum, OCC (Optimistic Concurrency Control), MVCC (Multiversion Concurrency Control) )

Blogs and Videos

• https://turbopuffer.com/blog/turbopuffer[turbopuffer: fast search on object storage]
• Uber ** https://www.uber.com/en-IN/blog/kafka-tiered-storage[Introduction to Kafka Tiered Storage at Uber]
• Redpanda ** https://docs.redpanda.com/current/manage/tiered-storage/[Redpanda Tiered Storage] ** https://www.redpanda.com/blog/tiered-storage-architecture-shadow-indexing-deep-dive[How we built Shadow Indexing to supercharge storage and data streaming]
• Substrate ** https://www.youtube.com/watch?v=9S8rmW8LD5o[Youtube, Deep Dive - Substrate Storage - Shawn Tabrizi]
• AWS ** https://www.allthingsdistributed.com/2024/08/continuous-reinvention-a-brief-history-of-block-storage-at-aws.html[Continuous reinvention: A brief history of block storage at AWS]

RFCs

• https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage[KIP-405: Kafka Tiered Storage]

== Data Consumption

Blogs

• https://medium.com/traveloka-engineering/data-lake-api-on-microservice-architecture-using-bigquery-10d6e9c5ca8f[Traveloka - Data Lake API on Microservice Architecture using BigQuery]. ** Best Practices? avoid giving direct access to data platform storage (object storage, database, etc.) as it creates a tight coupling to the underlying technology, format, etc. Instead, have an API layer in between to decouple that dependency. ** What's bad about direct access? *** change coordination required between teams. *** lack of access control (column, row levels). *** lack of audit log (who access, download what).
• Building Criteo API, What We’ve Learned (https://medium.com/criteo-engineering/building-criteo-api-what-weve-learned-b7f3e7b8d270). Key lessons learned after building a new API ecosystem from scratch.
• Idempotency Keys: How PayPal and Stripe Prevent Duplicate Payment(https://medium.com/@sahintalha1/the-way-psps-such-as-paypal-stripe-and-adyen-prevent-duplicate-payment-idempotency-keys-615845c185bf)
• How We Design Our APIs at Slack (https://slack.engineering/how-we-design-our-apis-at-slack/)
• Grafana - How I write HTTP services in Go after 13 years (https://grafana.com/blog/2024/02/09/how-i-write-http-services-in-go-after-13-years/)
• Introducing DoorDash’s In-House Search Engine (https://doordash.engineering/2024/02/27/introducing-doordashs-in-house-search-engine/)

== Data Processing Blogs

• https://medium.com/agoda-engineering/how-to-design-maintain-a-high-performing-data-pipeline-63b1603b8e4a[Agoda, How to Design and Maintain a High-Performing Data Pipeline] ** Data pipeline scalability: SLA, partioning, data freshness, resource usage, scheduling, data dependency, monitoring. ** Data quality: freshness, integrity (uniqueness e.g. no dup keys), completeness (e.g. no empty, NULLS), accuracy (value is not abnormal by checking with previous trend, ThridEye), consistency (source = destination, Quilliup, running when pipeline completes). ** Ensuring data quality: validating before writing to destination, testing, monitoring, alerting, responding, automatic Jira tickets creation.
• https://medium.com/criteo-engineering/scheduling-data-pipelines-at-criteo-part-1-8b257c6c8e55[Scheduling Data Pipelines at Criteo — Part 1]
• https://netflixtechblog.com/orchestrating-data-ml-workflows-at-scale-with-netflix-maestro-aaa2b41b800c[Orchestrating Data/ML Workflows at Scale With Netflix Maestro]
• Netflix's Dataflow: bootstrapping, standardization, automation of batch data pipelines ** https://netflixtechblog.com/data-pipeline-asset-management-with-dataflow-86525b3e21ca[Data pipeline asset management with Dataflow] ** https://netflixtechblog.com/ready-to-go-sample-data-pipelines-with-dataflow-17440a9e141d[Ready-to-go sample data pipelines with Dataflow] ** https://netflixtechblog.medium.com/etl-development-life-cycle-with-dataflow-9c70c64aba7b[ETL development lifecycle (testing and deployment) with Dataflow]
• Uber ** https://www.uber.com/en-IN/blog/no-code-workflow-orchestrator[uWorc: No Code Workflow Orchestrator for Building Batch & Streaming Pipelines at Scale, 2020] ** https://www.uber.com/en-GB/blog/sparkle-modular-etl/[Sparkle: Standardizing Modular ETL at Uber, 2024]

Papers

• Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine (https://github.com/apache/arrow-datafusion/issues/6782). Written in Rust, uses Apache Arrow as memory model.

Projects

• https://github.com/flyteorg/flyte: orchestrator

== Data Governance Metadata management, data quality, data veracity, data security, data lineage, etc.

• Open Metadata (https://open-metadata.org/)

=== Data Quality

Blogs

• How Google, Uber, and Amazon Ensure High-Quality Data at Scale
• Uber - Monitoring Data Quality at Scale with Statistical Modeling
• LinkedIn - Towards data quality management at LinkedIn
• Data Quality: Timeseries Anomaly Detection at Scale with Thirdeye
• How we deal with Data Quality using Circuit Breakers (https://medium.com/@modern-cdo/taming-data-quality-with-circuit-breakers-dbe550d3ca78)
• Lyft - From Big Data to Better Data: Ensuring Data Quality with Verity (https://eng.lyft.com/from-big-data-to-better-data-ensuring-data-quality-with-verity-a996b49343f6)
• Data Quality Automation at Twitter (https://blog.x.com/engineering/en_us/topics/infrastructure/2022/data-quality-automation-at-twitter)

Papers

• VLDB, Amazon - Automating Large-Scale Data Quality Verification. It presents the design choices and architecture of a production-grade system for checking data quality at scale, shows the evaluation result on some datasets.

Best Practices

• too little data quality alerts let important issues go unresolved.
• too many alerts overwhelms and might make the most important ones go un-noticed.
• statistical modeling techniques (PCA, etc.) can be used to reduce computation resource for quality checks.
• separate anomaly detection from anomaly scoring and alerting strategy.

Common Issues

• issues in metadata category (data availability, data freshness, schema changes, data completeness) -> can be obtained without checking dataset content
• issues in semantic category(dataset content: column value nullability, duplication, distribution, exceptional values, etc.) -> needs data profiling

=== Data Lineage Blogs

• Data Lineage at Slack (https://slack.engineering/data-lineage-at-slack/). ** Lineage service exposes endpoints for ingestion, stores data in RDS. ** Ingestion for Airflow DAGs built into existing dags using Airflow callbacks. ** Ingestion for Presto dashboards: audit tables, SQL Parsing.
• OpenLineage, open framework for data lineage collection and analysis (https://openlineage.io/)
• How we compute data lineage at Criteo (https://medium.com/criteo-engineering/how-we-compute-data-lineage-at-criteo-b3f09fc5c577)
• Yelp - Spark Data Lineage (https://engineeringblog.yelp.com/2022/08/spark-data-lineage.html)
• Data Lineage: State-of-the-art and Implementation Challenges (https://medium.com/bliblidotcom-techblog/data-lineage-state-of-the-art-and-implementation-challenges-1ea8dccde9de)

=== Data Security, Compliance GDPR, CCPA, PII Protection, etc.

• Lyft - A Federated Approach To Providing User Privacy Rights (https://eng.lyft.com/a-federated-approach-to-providing-user-privacy-rights-3d9ab73441d9). Technical strategies for CCPA. Implementation of user data export and deletion. Federated design with central orchestration for exporting/ deleting.
• Intuit - 10 lessons learned in operationalizing GDPR at scale (https://medium.com/ssdr-book/10-lessons-learned-in-operationalizing-gdpr-at-scale-7a41318846b6)

== Data Integration

Blogs

• How Agoda manages 1.8 trillion Events per day on Kafka (https://medium.com/agoda-engineering/how-agoda-manages-1-8-trillion-events-per-day-on-kafka-1d6c3f4a7ad1)
• Apache Kafka Rebalance Protocol, or the magic behind your streams applications (https://medium.com/streamthoughts/apache-kafka-rebalance-protocol-or-the-magic-behind-your-streams-applications-e94baf68e4f2)

== Machine Learning

• Featureflow: Democratizing ML for Agoda (https://medium.com/agoda-engineering/featureflow-democratizing-ml-for-agoda-aec7a6c45b30)
  • Challenge: time-consuming feature analysis, training, validation vs fast changing customers and competitors in travel industry; lacking of consistency from analysis to training, from feature development to deployment.
  • Solution: Featureflow with components (UI, data pipeline, monitoring, sandbox env, experiment platform)
  • Result: feature analysis reduced from a week to a day, quarterly experiments increased from 6 to 20, feature contributors from ~3 to ~50, larger feature pool, more robust feature screening process.
• How ByteDance Scales Offline Inference with multi-modal LLMs to 200 TB Data (https://www.anyscale.com/blog/how-bytedance-scales-offline-inference-with-multi-modal-llms-to-200TB-data)
• Building Real-time Machine Learning Foundations at Lyft (https://eng.lyft.com/building-real-time-machine-learning-foundations-at-lyft-6dd99b385a4e)

== Platform and Infra

Kubernetes

• Lessons From Our 8 Years Of Kubernetes In Production (https://medium.com/@.anders/learnings-from-our-8-years-of-kubernetes-in-production-two-major-cluster-crashes-ditching-self-0257c09d36cd)

Terraform

• Slack - How We Use Terraform At Slack (https://slack.engineering/how-we-use-terraform-at-slack/)

Network

• Slack - Traffic 101: Packets Mostly Flow (https://slack.engineering/traffic-101-packets-mostly-flow/)

== Testing

• Slack - Continuous Load Testing (https://slack.engineering/continuous-load-testing/)

=== Deterministic Simulation Testing (DST)

• https://apple.github.io/foundationdb/testing.html[FoundationDB - Testing. Where DST is made famous.]
• https://www.warpstream.com/blog/deterministic-simulation-testing-for-our-entire-saas[Warpstream - Deterministic Simulation Testing for Our Entire SaaS]
• https://www.polarsignals.com/blog/posts/2025/07/08/dst-rust[Polar Signals - Deterministic Simulation Testing in Rust: A Theater Of State Machines]
• https://docs.tigerbeetle.com/about/vopr/[TigerBeetle - Deterministic Simulation Testing]

Videos

• https://www.youtube.com/live/29Vz5wkoUR8[Youtube, FF meetup #4 - Deterministic simulation testing (Pekka Enberg, Turso)]
• https://www.youtube.com/watch?v=KoxXb2mBVuY[Youtube, Testing Distributed Systems the right way ft. Will Wilson (Antithesis)]

== Operations

Observability, Monitoring

• Observability @ Data Pipelines (https://medium.com/ssdr-book/observability-data-pipelines-99eda62b1704)

Incidents

• Slack’s Incident on 2022-Feb-22 (https://slack.engineering/slacks-incident-on-2-22-22/)

== Interview Resources

• https://medium.com/agoda-engineering/preparing-for-interview-at-agoda-2c07b7d13ca5[Preparing for Interview at Agoda]: interview process at Agoda with advices for candidates in each stage.
• https://gist.github.com/vasanthk/485d1c25737e8e72759f[System Design Cheatsheet]

== Databricks

Architecture

• A data architecture pattern to maximize the value of the Lakehouse (https://www.databricks.com/blog/data-architecture-pattern-maximize-value-lakehouse.html)

Data Pipelines

• How to Evaluate Data Pipelines for Cost to Performance (https://www.databricks.com/blog/2020/11/13/how-to-evaluate-data-pipelines-for-cost-to-performance.html)

Spark and Databricks Compute

• Advanced Topics on Spark Optimization and Debug (https://holdenk.github.io/spark-flowchart)
• Example Code for High Performance Spark book (https://github.com/high-performance-spark/high-performance-spark-examples)

Delta Lake

• Managing Recalls with Barcode Traceability on the Delta Lake
• Creating a Spark Streaming ETL pipeline with Delta Lake at Gousto
  • issues and solutions ** costly Spark op MSCK REPAIR TABLE because it needs to scan table' sub-tree in S3 bucket. -> use ALTER TABLE ADD PARTITION instead. ** not caching dataframes for multiple usages. -> use cache ** rewriting all destination table incl. old partitions when having a new partition. -> append new partition to destination. ** architecture (waiting for CI, Airflow triggering, EMR spinning up, job run, working with AWS console for logs) slowing down development. Min feedback loop of 20 minutes. -> move away from EMR, adopt a platform allowing to have complete control of clusters and prototyping.
  • Databricks Pros ** Reducing ETL time, latency from 2 hours to 15s by using streaming job and delta architecture. ** Spark Structured Streaming Autoloader helps manage infra (setting up bucket noti, SNS and SQS in the background). ** Notebook helps prototype on/ explore production data, debug with traceback and logs interactively. Then CICD to deploy when code is ready. This helps reduce dev cycle from 20 mins to seconds. ** Costs remain the same as before Databricks. (using smaller instances with streaming cluster, which compensated for DBx higher costs vs EMR). ** Reducing complexity in codebase and deployment (no Airflow). ** Better ops: performance dashboards, Spark UI, reports.
  • Other topics: DBT for data modeling, Redshift, SSOT.
• Data Modeling Best Practices & Implementation on a Modern Lakehouse

Governance

• https://www.databricks.com/blog/2022/03/23/implementing-the-gdpr-right-to-be-forgotten-in-delta-lake.html[Implementing the GDPR 'Right to be Forgotten' in Delta Lake.] Approaches: 1-Data Amnesia, 2-Anonymization, 3-Pseudonymization/Normalized tables. Sped up point DELETE by data skipping optimization with Z-order on DELETE where fields.

Backfilling

• https://docs.databricks.com/en/ingestion/auto-loader/production.html=trigger-regular-backfills-using-cloudfilesbackfillinterval
• https://community.databricks.com/t5/data-engineering/how-to-make-structured-streaming-with-autoloader-efficiently-and/td-p/47833
• https://community.databricks.com/t5/data-engineering/autoloader-start-and-end-date-for-ingestion/td-p/45523[Autoloader start and end date for ingestion]

== Performance

Papers

• https://www.usenix.org/conference/atc18/presentation/yang-stephen[NanoLog: A Nanosecond Scale Logging System]. https://github.com/sonhmai/NanoLog[github repo]. Implementing a fast, low latency, high thruput C++ logging system by 1-shifting work out of the runtime hot path and into the compilation and post-execution phases of the application, 2-deferring formatting to an offline process.

== Databases

• https://transactional.blog/blog/2024-database-startups[2024 Database startups]
• https://transactional.blog/blog/2025-decomposing-transactional-systems[Decomposing Transactional Systems] ** https://brooker.co.za/blog/2025/04/17/decomposing.html[Decomposing Aurora DSQL] ** https://www.firebolt.io/blog/decomposing-firebolt-transactions[Decomposing Firebolt transactions]

=== LSM Tree Storage Engine

• https://artem.krylysov.com/blog/2023/04/19/how-rocksdb-works/[How Rocksdb Works]
• https://www.youtube.com/watch?v=23OmqULDld0[SREcon22 - Dissecting the Humble LSM Tree and SSTable]. In this talk on YouTube, we will start from the ground up, describing what the LSM data structure is and implementing it using code. We'll then talk about the operational characteristics and how they are used in practice for high performance applications that need to store data.
• https://www.youtube.com/watch?v=32LMicc0gRA[Redesigning OLTP for a New Order of Magnitude]. A InfoQ talk where Joran Greef discusses TigerBeetle, a new database, and why OLTP has a growing impedance mismatch, why the OLTP workload is becoming more contentious, why row locks, why storage faults, write stalls, and why non-determinism is now a problem.

== Serverless Databases

Collections of resources of how serverless databases work

=== Papers

1. https://dx.doi.org/10.1145/3035918.3056101[A. Verbitski et al., “Amazon Aurora,” Proceedings of the 2017 ACM International Conference on Management of Data.]
2. P. Antonopoulos et al., “Socrates,” Proceedings of the 2019 International Conference on Management of Data.
3. W. Cao et al., “PolarDB Serverless,” Proceedings of the 2021 International Conference on Management of Data.

=== Blogs

1. https://neon.tech/blog/how-we-scale-an-open-source-multi-tenant-storage-engine-for-postgres-written-rust[Neon. How we scale an open source, multi-tenant storage engine for Postgres written in Rust].