CUDA Programming from Scratch

CUDA Programming from Scratch

1. From First Principles to Production-Grade GPU Applications

Introduction: Why GPU Computing Matters Today

Chapter 1: The GPU Revolution — Architecture and History

1. From Graphics to General-Purpose Computing
2. GPU vs CPU: Divergent Design Philosophies
3. The CUDA Platform Ecosystem
4. GPU Architecture Roadmap: Fermi through Blackwell

Chapter 2: The CUDA Programming Model — Threads, Blocks, and Warps

1. SIMT Execution: Single Instruction, Multiple Threads
2. Thread Hierarchy: Threads, Warps, Blocks, Grids, and Clusters
3. Kernel Launch Syntax and Configuration
4. The Grid-Stride Loop Pattern
5. Occupancy: Theory, Calculation, and the Occupancy API

Chapter 3: Memory Models — The GPU Memory Hierarchy

1. Registers and Local Memory
2. Global Memory and Coalesced Access
3. Shared Memory: Scope, Latency, and Bank Conflicts
4. Constant and Texture Memory
5. L1/L2 Cache Architecture
6. Memory Alignment and Vectorized Access

Chapter 4: Writing and Optimizing Kernels

1. Your First CUDA Kernels: Vector Addition, Matrix Multiply
2. Tiling and Shared Memory Optimization
3. Avoiding Bank Conflicts: Padding and Swizzling
4. Warp-Specialized Kernels and Producer-Consumer Patterns
5. Common Pitfalls: Divergence, Race Conditions, Out-of-Bounds

Chapter 5: Synchronization — From Warps to Grids

1. Warp-Level Synchronization (Implicit and Explicit)
2. Block-Level Barriers (syncthreads)
3. Cooperative Groups: Thread Block Tiles, Cluster Groups, Grid Groups
4. Scoped Atomics and Thread Scopes
5. Asynchronous Barriers and cuda::barrier (Hopper+)

Chapter 6: Warp-Level and Intrinsics Programming

1. Warp Shuffle Primitives: __shfl_sync, __shfl_down_sync, __shfl_up_sync, __shfl_xor_sync
2. Vote and Mask Operations: __ballot_sync, __any_sync, __all_sync, __activemask
3. Warp-Level Reductions and Scans
4. Inline PTX Assembly for Performance-Critical Code
5. When to Use Warp Primitives vs. Cooperative Groups

Chapter 7: Asynchronous Execution — Streams, Events, and Overlap

1. CUDA Streams: Default and User-Created
2. Events for Synchronization and Timing
3. Overlapping Data Transfers with Computation
4. Multi-Stream Pipelining Patterns
5. CUDA Graphs: Capture, Replay, and Constant-Time Launch

Chapter 8: Unified Memory and Advanced Memory Management

1. The Problem with Explicit Host-Device Transfers
2. cudaMallocManaged and Page Migration Engine
3. Pinned (Page-Locked) Memory
4. Unified Memory Performance: When It Works, When It Doesn’t

Chapter 9: Tensor Cores and Mixed-Precision Computing

1. Evolution of Tensor Cores: Volta through Blackwell
2. Matrix Multiply-Accumulate (MMA) Operations
3. Data Precision Formats
4. Writing Tensor Core Kernels with WGMMA
5. The Transformer Engine and Dynamic Scaling

Chapter 10: Hopper Innovations — TMA, Barriers, and Pipelines

1. Tensor Memory Accelerator (TMA): Architecture and Programming Model
2. cuda::memcpy_async and Asynchronous Data Copies
3. CUDA Pipelines: Producer-Consumer Patterns with Multi-Buffering
4. Warp-Specialized Kernels for Maximum Utilization
5. Cluster-Sized Thread Blocks and GPC-Level Scheduling

Chapter 11: CUDA Libraries — Building on NVIDIA’s Foundation

1. Linear Algebra: cuBLAS, cuSOLVER, cuSPARSE
2. Signal Processing: cuFFT
3. Parallel Primitives: CUB and Thrust
4. Random Numbers: cuRAND
5. Image and Video: NPP, nvJPEG, nvCodec
6. When to Use Libraries vs. Custom Kernels

Chapter 12: Dynamic Parallelism and Multi-GPU Programming

1. Dynamic Parallelism: Child Kernels, Nested Launches
2. Multi-GPU Architecture: PCIe vs NVLink
3. Peer-to-Peer Memory Access (GPUDirect P2P)
4. NCCL: Collective Communication Primitives
5. Multi-GPU Design Patterns and Scaling Considerations

Chapter 13: Profiling, Debugging, and Performance Engineering

1. CUDA-GDB and Nsight Debugger
2. Compute Sanitizer: memcheck, racecheck, synccheck, initcheck
3. Nsight Systems: System-Wide Profiling
4. Nsight Compute: Kernel-Level Metrics and Analysis
5. The APOD Framework (Assess, Parallelize, Optimize, Deploy)
6. Performance Engineering Case Studies

Chapter 14: Real-World Applications — AI, HPC, and Scientific Computing

1. Convolutional Kernels for Image Processing
2. Matrix Multiplication at Scale: From Naive to Tensor Core Optimized
3. Sparse Linear Algebra for Scientific Computing
4. AI Training and Inference Pipelines
5. Mini-Project: GPU-Accelerated Particle Simulation

Conclusion: The Future of GPU Computing

References