Skip to content
Projects

goabiaryan/awesome-gpu-engineering

View on GitHub

goabiaryan/awesome-gpu-engineering has 610 GitHub stars and 78 forks. GitHot measured +1 stars in 24 hours. Last checked 2026-11-15 16:37 UTC. Last push 2026-05-17.

How these counts are measured

Measured star growth

in 24h
+1
in 7 days
Update delayed
in 15 days
Update delayed
in 1 month
Update delayed

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

GPU Engineering for AI Systems

Learning

README.md

Original

Awesome GPU Engineering Awesome

A curated list of resources for mastering GPU engineering from architecture and kernel programming to large-scale distributed systems and AI acceleration.


📘 Foundational Books

  • Programming Massively Parallel Processors: A Hands-on Approach — David B. Kirk & Wen-mei W. Hwu The canonical introduction to CUDA, memory hierarchies, and parallel patterns. Amazon , notes: Abi's Concise Notes
  • CUDA by Example — Jason Sanders & Edward Kandrot
    A practical introduction to CUDA for beginners. Amazon
  • The Ultra-Scale Playbook: Training LLMs on GPU Clusters - Hugging Face Web Version

💻 GPU Programming Frameworks

  • CUDA — NVIDIA’s proprietary GPU programming platform.
  • ROCm — AMD’s open compute stack.
  • OpenCL — Cross-platform parallel computing standard.
  • SYCL / oneAPI — Intel’s C++ abstraction for heterogeneous compute.
  • Vulkan Compute — Low-level GPU compute API.
  • Kompute — Higher level general purpose GPU compute framework built on Vulkan.
  • Metal Performance Shaders — Apple’s GPU framework.
  • Mojo🔥 - Write like Python, run like C++.

🧩 Optimization and Performance

  • NVIDIA Nsight Systems — System-wide GPU profiler.
  • Nsight Compute — Kernel-level performance analysis.
  • Occupancy Calculator — NVIDIA spreadsheet for kernel configuration.
  • CUTLASS — CUDA templates for linear algebra subroutines.
  • TensorRT — High-performance deep learning inference.
  • OpenAI Triton — Python DSL for writing high-performance GPU kernels.
  • Helion - A Python-embedded DSL that makes it easy to write fast, scalable ML kernels with minimal boilerplate.
  • Roofline Model — Analytical model to reason about compute/memory bottlenecks.

🧠 Architecture and Low-Level Design

⚙️ Systems and Multi-GPU Engineering

🧪 Tutorials and Courses

📄 Research Papers and Articles

🧰 Tools and Utilities

  • nvprof, nvvp, Nsight Systems / Compute — NVIDIA profiling tools.
  • cuda-memcheck, compute-sanitizer — Memory and correctness tools.
  • GPGPU-Sim, Accel-Sim — GPU simulation frameworks.
  • Ingero — eBPF-based GPU causal observability agent. Traces CUDA Runtime/Driver APIs and host kernel events to build causal chains explaining GPU latency. <2% overhead, production-safe.
  • Perfetto, Nsight UI — Visual profilers for tracing GPU workloads.

Learning Tools

🧑‍🔬 GPU for AI & ML

  • PyTorch CUDA Extensions — Custom kernels for PyTorch.
  • JAX + XLA — Compiler-based GPU vectorization.
  • TensorFlow XLA Compiler — Ahead-of-time GPU graph compilation.
  • FlashAttention, FlashConv — Kernel optimization techniques for transformers.
  • DeepSpeed, FSDP, Megatron-LM — Distributed training systems.

🧱 GPU Systems Design Topics For Interview Prep

  • FlashAttention and PagedAttention
  • Matmul Operations
  • GPU scheduling algorithms and runtime systems.
  • Memory oversubscription and unified memory models.
  • Resource allocation in GPU clusters.
  • GPU virtualization
  • Kernel fusion and graph execution
  • Dataflow optimization
  • Persistent threads model

🧑‍💻 Contributors

Contributions welcome! Please read the contribution guidelines before submitting a pull request.

🧾 License

CC BY 4.0 — feel free to share and adapt with attribution.

⭐ Acknowledgements

Inspired by:


“GPU engineering is not just about writing kernels. It’s about understanding how systems work.” — Model Craft