The Leanpub 60 Day 100% Happiness Guarantee
Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.
See full terms...
An Advanced Engineering Reference for GPU Architecture, Low-Level Optimization, and High-Performance Computing
Modern GPU programming requires more than exposing parallel work to thousands of threads. It demands an understanding of the relationship between hardware architecture, instruction execution, memory movement, synchronization, and the software systems that coordinate computation across multiple GPUs. Mastering NVIDIA CUDA: Expert GPU Programming is a 100-chapter technical reference for experienced developers, high-performance computing engineers, researchers, and AI systems architects.
Minimum price
$31.00
$31.00
Buying multiple copies for your team? See below for a discount!
About the Book
An Advanced Engineering Reference for GPU Architecture, Low-Level Optimization, and High-Performance Computing
Modern GPU programming requires more than exposing parallel work to thousands of threads. It demands an understanding of the relationship between hardware architecture, instruction execution, memory movement, synchronization, and the software systems that coordinate computation across multiple GPUs. Mastering NVIDIA CUDA: Expert GPU Programming is a 100-chapter technical reference for experienced developers, high-performance computing engineers, researchers, and AI systems architects. It explores GPU performance engineering from microarchitecture and low-level machine code to distributed computing, production deployment, and specialized computational workloads, with particular attention to NVIDIA Hopper and Blackwell systems. The book moves beyond introductory CUDA programming to examine how hardware constraints influence software design. Topics include instruction-level parallelism, register pressure, warp scheduling, memory hierarchy, asynchronous execution, mixed-precision arithmetic, and multi-GPU communication. The goal is to help readers reason about performance at the level where implementation choices meet the capabilities and limitations of the hardware.
Understanding GPU performance begins with understanding what the hardware actually executes. The book examines Streaming Multiprocessor organization, register allocation, warp-level synchronization, control-flow divergence, predication, and shared-memory bank conflicts.
Dedicated chapters explore PTX as CUDA's virtual instruction set and SASS as NVIDIA's machine-level instruction representation. Profiling, instruction analysis, occupancy, and performance modeling are treated as complementary tools for identifying bottlenecks rather than relying on assumptions about what should be fast.
The discussion spans architectural developments from Fermi through Hopper and Blackwell, providing historical context for changes in execution models, memory systems, and specialized compute capabilities.
GPU performance often depends as much on moving data efficiently as on performing arithmetic. This reference examines shared-memory pipelines, cache behavior, memory allocation, Unified Memory, stream-ordered operations, and asynchronous data movement.
Hopper and Blackwell topics include the Tensor Memory Accelerator (TMA), thread block clusters, Distributed Shared Memory, and synchronization mechanisms for coordinating work across Streaming Multiprocessors. Double-buffering, data-layout design, and memory-access optimization are considered in the context of practical kernel engineering.
The book also explores CUDA Graphs, cooperative execution, and other techniques for reducing host-side launch overhead and coordinating complex GPU workloads.
Modern AI workloads depend on specialized arithmetic units, carefully designed data layouts, and efficient movement of model parameters and intermediate results. The book covers Tensor Core programming, mixed-precision computation, FP8 and FP4 formats, custom matrix multiplication kernels, quantization, and kernel fusion.
Additional chapters address TensorRT inference, custom AI plugins, data-loading pipelines, and GPU acceleration for large language models. The emphasis is on understanding the trade-offs among throughput, numerical behavior, memory consumption, and latency.
Scaling beyond a single GPU introduces communication costs and coordination challenges that cannot be solved through kernel optimization alone.
The book examines NVLink and NVSwitch interconnects, GPUDirect RDMA, GPUDirect Storage, NCCL communication patterns, collective operations, and multi-GPU workload distribution. It also explores all-reduce and all-to-all operations, CUDA streams and events, and the architectural considerations involved in building larger GPU compute systems.
These topics connect individual GPU programming techniques to the broader requirements of distributed AI training, scientific computing, and high-performance cluster workloads.
Performance engineering requires measurable evidence. Chapters on NVIDIA Nsight Systems, Nsight Compute, Roofline analysis, debugging, sanitizers, and race-condition detection provide a framework for investigating execution behavior and validating optimization decisions.
Application-oriented material explores sparse matrix computations, FFTs, graph analytics, Monte Carlo simulation, computational fluid dynamics, molecular dynamics, computer vision, video processing, signal processing, cryptographic workloads, and scientific visualization.
Further topics include Python integration through CuPy and Numba, CUDA interoperability with Rust and modern C++, C++ standard parallelism, OpenMP offloading, and performance-portable approaches using Kokkos and RAJA.
Advanced GPU software must remain reliable outside controlled benchmarks. The final sections examine formal verification, static analysis, memory validation, security vulnerabilities, energy efficiency, thermal throttling, custom memory allocators, and the CUDA Driver API and Runtime API. Production engineering topics include CI/CD pipelines, Kubernetes and Slurm deployment, long-running GPU services, and sustained performance under operational constraints. The book also considers the Grace Hopper platform, heterogeneous memory systems, and emerging directions in GPU computing.
This reference is intended for experienced CUDA developers optimizing existing applications, HPC engineers working with large-scale simulations, researchers developing computational kernels, and machine learning systems engineers seeking better performance from modern GPU hardware.
It is particularly suited to readers who already understand programming fundamentals and want to deepen their knowledge of GPU architecture, low-level execution, profiling, memory behavior, and distributed compute systems. The material is technical in scope and is best approached as an advanced reference rather than a first introduction to CUDA.
The bundle combines the core textbook with supplementary reference materials intended to support implementation, review, and further study.
Core Textbook — PDF and TXT contains the 100-chapter reference covering GPU architecture, low-level optimization, parallel algorithms, and production systems. A sample chapter provides a preview of the book's technical approach.
The Blackwell CUDA Blueprint presents architectural diagrams and a technical overview of Blackwell-related design concepts and GPU interconnects. NVIDIA CUDA Architecture Features & Optimization Techniques provides a comparative reference across GPU generations, while CUDA & Blackwell Architecture Executive Briefing summarizes performance models and scaling considerations.
Advanced GPU Compute Fabric Architectures supplies a high-resolution architectural schematic covering GPU components such as Graphics Processing Clusters, Texture Processing Clusters, and Streaming Multiprocessors.
The bundle also includes TMA: Zero-Overhead Data Movement, a technical video exploring asynchronous transfers and synchronization, together with an Interactive Study Engine containing 79 structured flashcards for reviewing PTX, SASS, memory layouts, and consistency concepts.
High-performance GPU programming is not a matter of applying a single optimization technique. It is the process of understanding the workload, identifying the limiting resource, selecting an appropriate execution strategy, and validating the result against real hardware behavior. Mastering NVIDIA CUDA: Expert GPU Programming brings these concerns together in a broad, systems-oriented reference for developers working at the intersection of GPU architecture, scientific computing, and modern AI infrastructure.
Table of Contents
1. The Evolution of Compute: From Fermi to Blackwell
2. The Blackwell Microarchitecture: A Paradigm Shift in Throughput
3. Hopper Architecture Foundations: TPCs, GPCs, and SMs
4. The Physics of Modern GPU Hardware Constraints
5. Advanced Thread Block Cluster Orchestration
6. Mastering the Tensor Memory Accelerator
7. L2 Cache Residency and Persistence Control
8. High Bandwidth Memory 3e: Latency vs Throughput
9. The Anatomy of a Modern Streaming Multiprocessor
10. Register Pressure Management and Spilling Strategies
11. Advanced Warp-Level Primitives and Synchronization
12. PTX Deep Dive: Programming the Virtual Machine
13. Mastering SASS: Reading and Optimizing Machine Code
14. Control Flow Divergence and Predication Optimization
15. Shared Memory Bank Conflicts and Padding in the Blackwell Era
16. Distributed Shared Memory in Hopper and Blackwell
17. Cooperative Groups for Heterogeneous Workoads
18. Asynchronous Data Movement with Copy Async
19. Transactional Memory and Atomic Operations at Scale
20. Designing for High Instruction-Level Parallelism
21. Exploiting Thread-Level Parallelism Limits
22. Tensor Cores: 4th and 5th Generation Deep Dive
23. Mixed-Precision Arithmetic: Mastering FP8 and FP4
24. The Transformer Engine: Hardware-Accelerated Attention
25. Implementing Custom Matrix Multiply-Accumulate Kernels
26. CUDA Graphs: Eliminating CPU-side Overhead
27. Stream-Ordered Memory Allocation and Management
28. Multi-Instance GPU for High-Density Computing
29. Confidential Computing on H100 and B200
30. NVLink 5: High-Speed Interconnect Topologies
31. NVSwitch Fabric: Designing Scalable GPU Clusters
32. GPUDirect RDMA and Storage Technologies
33. Collective Communications: NCCL Internal Mechanics
34. Optimizing All-Reduce and All-to-All Operations
35. Pipelining Workloads with CUDA Streams and Events
36. JIT Compilation and Runtime Kernel Generation
37. Link-Time Optimization in CUDA Applications
38. Advanced Debugging with NVIDIA Nsight Systems
39. Profiling Kernels with Nsight Compute: Metrics and Baselines
40. Roofline Analysis for Performance Bottleneck Identification
41. Managing Occupancy and Warp Schedulers
42. High-Performance Sparse Matrix Computations
43. Fast Fourier Transforms at Exascale
44. Graph Analytics and Irregular Parallelism
45. Accelerating Deep Learning Inference with TensorRT
46. Custom Layers and Plugin Development for AI
47. Dynamic Programming and Sequence Alignment on GPUs
48. Monte Carlo Simulations and RNG Optimization
49. Cryptographic Primitives and Blockchain Acceleration
50. CFD and Molecular Dynamics: Real-world HPC Scaling
51. Ray Tracing Cores for Non-Graphics Workloads
52. Video Encoding and Decoding via NVENC and NVDEC
53. Optical Flow and Computer Vision Hardware Acceleration
54. Unified Memory: Advanced Prefetching and Advice
55. Heterogeneous Memory Management on Linux
56. Multi-GPU Load Balancing and Task Distribution
57. Zero-Copy Memory and PCIe Gen 6 Considerations
58. Building High-Performance Data Loaders for AI
59. Quantization Algorithms for Large Language Models
60. Kernel Fusion Strategies for Complex Pipelines
61. Autotuning Kernels for Cross-Architecture Portability
62. The Future of GPGPU: Beyond Blackwell
63. Hardware-Software Co-Design for Specialized Domains
64. Implementing Software-Managed Caches in Shared Memory
65. Bit-Manipulation Instructions and Advanced Integer Math
66. Handling Large Scale Data with NVIDIA Multi-Node Toolkit
67. Virtual Memory Management API Deep Dive
68. Cooperative Launch and Grid-Level Synchronization
69. Reducing Tail Latency in Real-time GPU Systems
70. Optimizing for the NVIDIA Grace Hopper Superchip
71. The Grace CPU and LPDDR5X Memory Integration
72. C++ Standard Parallelism for NVIDIA GPUs
73. Python Performance with CuPy and Numba
74. Integrating CUDA with Rust and Modern Systems Languages
75. Formal Verification of CUDA Kernels
76. Security Vulnerabilities in GPGPU Kernels
77. Energy-Efficient GPU Programming
78. Managing Thermal Throttling and Power Limits
79. CI/CD Pipelines for High-Performance Computing
80. Deploying CUDA at Scale: Kubernetes and Slurm
81. Debugging Race Conditions and Memory Corruption
82. Using Sanitizers for CUDA Memory Validation
83. Implementing Custom Allocators for GPU Memory
84. Exploring the CUDA Driver API vs Runtime API
85. Static Analysis Tools for Parallel Code
86. Performance Portability with Kokkos and RAJA
87. OpenMP Offloading for NVIDIA Hardware
88. The CUTLASS Library: Templates for Matrix Multiplies
89. CuTe: A New Abstraction for Tiled Layouts
90. NVIDIA Warp: A Framework for Physics Simulation
91. Optimizing Stencil Computations and Finite Difference
92. Sort Scan and Reduction: The Building Blocks of Parallelism
93. Designing Low-Latency Audio and Signal Processing Kernels
94. Exploiting Hardware-Based Data Compression
95. Case Study: Optimizing Llama-3 for Blackwell
96. Case Study: Climate Modeling at Petabyte Scales
97. Case Study: Real-time Ray Traced Scientific Visualization
98. Sustaining Performance in Long-Running GPU Services
99. Community Ecosystem and Staying Current with CUDA Research
100. Masterclass Conclusion: The Path to Zero-Overhead Computing
Team Discounts
Get a team discount on this book!
Up to 3 members
Up to 5 members
Up to 10 members
Up to 15 members
Up to 25 members
About the Author
I am an independent technology developer and systems engineer who built my technical path largely through self-directed engineering, experimentation, and continuous learning outside a traditional academic or corporate technology career.
My professional background began far from the technology industry. I spent years working in manufacturing, while independently developing my knowledge of software engineering, computer systems, and advanced computing. Over time, that self-directed work evolved into a broad technical practice spanning autonomous AI, cybersecurity, systems programming, GPU computing, automation, and advanced computational architectures.
Today, I design, build, and publish projects involving agentic AI, autonomous defense systems, SIEM/EDR integration, secure software architecture, C/C++, Go, Python, CUDA, quantum computing, cryptography, and privacy-oriented local AI infrastructure.
I approach technology from a systems perspective — from low-level software, memory architecture, and GPU performance to distributed systems, intelligent agents, and high-assurance security architectures.
I also explore aerospace and high-assurance software concepts, including safety-critical architectures, multi-level security, cross-domain solutions, and advanced computational systems.
Alongside active development, I publish long-form engineering projects covering AI, cybersecurity, cloud engineering, quantum computing, GPU programming, cryptography, automation, blockchain, and aerospace engineering.
My current focus is on autonomous software agents, privacy-first local infrastructure, advanced computing, and reliable systems designed to operate with a high degree of independence.
I am open to opportunities involving AI engineering, cybersecurity, software engineering, autonomous systems, HPC/GPU computing, and advanced technology development.
https://businessofmachines.blogspot.com/
https://learn.microsoft.com/en-us/users/machinadeusex/
Click the buttons to get the free sample in PDF or EPUB, or read the sample online here
Also by the Author
Linux GPU Drivers from Scratch
Mastering LLM Quantization: Architecture and Code
Engineering Sovereign LLMs: From Data to Deployment
CUDA C++: High-Performance GPU Programming
Agentic Cybersecurity: Engineering Autonomous AI Defense
Blockchain Security: Attack and Defense
The Local AI Stack: Building a Sovereign Machine Learning Workstation with Hyper-V, WSL2, and GPU Virtualization
The Solana MEV Playbook: Building High-Frequency Arbitrage and MEV Bots in Rust
The Quantization Black Book
Mastering ELF: From Bytes to Execution
Building a Native LLM Inference Engine
Modern Browser Fingerprinting: Techniques and Code
Mastering AI Agents: Hands-On Production Code
Coding CERN: A Technical Developer's Guide
High-Performance Real-Time Systems in .NET
Building Advanced Algorithmic Crypto Trading Systems
Smart Contract Security: Finding DeFi Vulnerabilities
Advanced Aerospace Engineering: Lockheed & Partners
Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.
See full terms...
We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.
(Yes, some authors have already earned much more than that on Leanpub.)
Learn more about writing on Leanpub
If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).
Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.
Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.
Learn more about Leanpub's ebook formats and where to read them
You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!
Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.
Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.