Leanpub Header

Skip to main content

Mastering NVIDIA CUDA: Expert GPU Programming

Mastering NVIDIA CUDA: Expert GPU Programming
This book is 100% completeLast updated on 2026-08-27

An Advanced Engineering Reference for GPU Architecture, Low-Level Optimization, and High-Performance Computing

Modern GPU programming requires more than exposing parallel work to thousands of threads. It demands an understanding of the relationship between hardware architecture, instruction execution, memory movement, synchronization, and the software systems that coordinate computation across multiple GPUs. Mastering NVIDIA CUDA: Expert GPU Programming is a 100-chapter technical reference for experienced developers, high-performance computing engineers, researchers, and AI systems architects.

Minimum price

$31.00

$31.00

You pay

Author earns

$

Also available for 1 book credit with a Reader Membership

Buying multiple copies for your team? See below for a discount!

PDF
351
Pages
Discussion Forum
About

About

About the Book

Mastering NVIDIA CUDA: Expert GPU Programming

An Advanced Engineering Reference for GPU Architecture, Low-Level Optimization, and High-Performance Computing

Modern GPU programming requires more than exposing parallel work to thousands of threads. It demands an understanding of the relationship between hardware architecture, instruction execution, memory movement, synchronization, and the software systems that coordinate computation across multiple GPUs. Mastering NVIDIA CUDA: Expert GPU Programming is a 100-chapter technical reference for experienced developers, high-performance computing engineers, researchers, and AI systems architects. It explores GPU performance engineering from microarchitecture and low-level machine code to distributed computing, production deployment, and specialized computational workloads, with particular attention to NVIDIA Hopper and Blackwell systems. The book moves beyond introductory CUDA programming to examine how hardware constraints influence software design. Topics include instruction-level parallelism, register pressure, warp scheduling, memory hierarchy, asynchronous execution, mixed-precision arithmetic, and multi-GPU communication. The goal is to help readers reason about performance at the level where implementation choices meet the capabilities and limitations of the hardware.

Microarchitecture, PTX, and SASS

Understanding GPU performance begins with understanding what the hardware actually executes. The book examines Streaming Multiprocessor organization, register allocation, warp-level synchronization, control-flow divergence, predication, and shared-memory bank conflicts.

Dedicated chapters explore PTX as CUDA's virtual instruction set and SASS as NVIDIA's machine-level instruction representation. Profiling, instruction analysis, occupancy, and performance modeling are treated as complementary tools for identifying bottlenecks rather than relying on assumptions about what should be fast.

The discussion spans architectural developments from Fermi through Hopper and Blackwell, providing historical context for changes in execution models, memory systems, and specialized compute capabilities.

Memory Movement and Asynchronous Execution

GPU performance often depends as much on moving data efficiently as on performing arithmetic. This reference examines shared-memory pipelines, cache behavior, memory allocation, Unified Memory, stream-ordered operations, and asynchronous data movement.

Hopper and Blackwell topics include the Tensor Memory Accelerator (TMA), thread block clusters, Distributed Shared Memory, and synchronization mechanisms for coordinating work across Streaming Multiprocessors. Double-buffering, data-layout design, and memory-access optimization are considered in the context of practical kernel engineering.

The book also explores CUDA Graphs, cooperative execution, and other techniques for reducing host-side launch overhead and coordinating complex GPU workloads.

Tensor Cores, Mixed Precision, and AI Systems

Modern AI workloads depend on specialized arithmetic units, carefully designed data layouts, and efficient movement of model parameters and intermediate results. The book covers Tensor Core programming, mixed-precision computation, FP8 and FP4 formats, custom matrix multiplication kernels, quantization, and kernel fusion.

Additional chapters address TensorRT inference, custom AI plugins, data-loading pipelines, and GPU acceleration for large language models. The emphasis is on understanding the trade-offs among throughput, numerical behavior, memory consumption, and latency.

Multi-GPU Fabrics and Distributed Computing

Scaling beyond a single GPU introduces communication costs and coordination challenges that cannot be solved through kernel optimization alone.

The book examines NVLink and NVSwitch interconnects, GPUDirect RDMA, GPUDirect Storage, NCCL communication patterns, collective operations, and multi-GPU workload distribution. It also explores all-reduce and all-to-all operations, CUDA streams and events, and the architectural considerations involved in building larger GPU compute systems.

These topics connect individual GPU programming techniques to the broader requirements of distributed AI training, scientific computing, and high-performance cluster workloads.

Profiling, Scientific Computing, and Real-World Applications

Performance engineering requires measurable evidence. Chapters on NVIDIA Nsight Systems, Nsight Compute, Roofline analysis, debugging, sanitizers, and race-condition detection provide a framework for investigating execution behavior and validating optimization decisions.

Application-oriented material explores sparse matrix computations, FFTs, graph analytics, Monte Carlo simulation, computational fluid dynamics, molecular dynamics, computer vision, video processing, signal processing, cryptographic workloads, and scientific visualization.

Further topics include Python integration through CuPy and Numba, CUDA interoperability with Rust and modern C++, C++ standard parallelism, OpenMP offloading, and performance-portable approaches using Kokkos and RAJA.

Verification, Security, and Production Deployment

Advanced GPU software must remain reliable outside controlled benchmarks. The final sections examine formal verification, static analysis, memory validation, security vulnerabilities, energy efficiency, thermal throttling, custom memory allocators, and the CUDA Driver API and Runtime API. Production engineering topics include CI/CD pipelines, Kubernetes and Slurm deployment, long-running GPU services, and sustained performance under operational constraints. The book also considers the Grace Hopper platform, heterogeneous memory systems, and emerging directions in GPU computing.

Who This Book Is For

This reference is intended for experienced CUDA developers optimizing existing applications, HPC engineers working with large-scale simulations, researchers developing computational kernels, and machine learning systems engineers seeking better performance from modern GPU hardware.

It is particularly suited to readers who already understand programming fundamentals and want to deepen their knowledge of GPU architecture, low-level execution, profiling, memory behavior, and distributed compute systems. The material is technical in scope and is best approached as an advanced reference rather than a first introduction to CUDA.

The Complete Technical Bundle

The bundle combines the core textbook with supplementary reference materials intended to support implementation, review, and further study.

Core Textbook — PDF and TXT contains the 100-chapter reference covering GPU architecture, low-level optimization, parallel algorithms, and production systems. A sample chapter provides a preview of the book's technical approach.

The Blackwell CUDA Blueprint presents architectural diagrams and a technical overview of Blackwell-related design concepts and GPU interconnects. NVIDIA CUDA Architecture Features & Optimization Techniques provides a comparative reference across GPU generations, while CUDA & Blackwell Architecture Executive Briefing summarizes performance models and scaling considerations.

Advanced GPU Compute Fabric Architectures supplies a high-resolution architectural schematic covering GPU components such as Graphics Processing Clusters, Texture Processing Clusters, and Streaming Multiprocessors.

The bundle also includes TMA: Zero-Overhead Data Movement, a technical video exploring asynchronous transfers and synchronization, together with an Interactive Study Engine containing 79 structured flashcards for reviewing PTX, SASS, memory layouts, and consistency concepts.

Engineering for Measurable Performance

High-performance GPU programming is not a matter of applying a single optimization technique. It is the process of understanding the workload, identifying the limiting resource, selecting an appropriate execution strategy, and validating the result against real hardware behavior. Mastering NVIDIA CUDA: Expert GPU Programming brings these concerns together in a broad, systems-oriented reference for developers working at the intersection of GPU architecture, scientific computing, and modern AI infrastructure.

Table of Contents

1. The Evolution of Compute: From Fermi to Blackwell

2. The Blackwell Microarchitecture: A Paradigm Shift in Throughput

3. Hopper Architecture Foundations: TPCs, GPCs, and SMs

4. The Physics of Modern GPU Hardware Constraints

5. Advanced Thread Block Cluster Orchestration

6. Mastering the Tensor Memory Accelerator

7. L2 Cache Residency and Persistence Control

8. High Bandwidth Memory 3e: Latency vs Throughput

9. The Anatomy of a Modern Streaming Multiprocessor

10. Register Pressure Management and Spilling Strategies

11. Advanced Warp-Level Primitives and Synchronization

12. PTX Deep Dive: Programming the Virtual Machine

13. Mastering SASS: Reading and Optimizing Machine Code

14. Control Flow Divergence and Predication Optimization

15. Shared Memory Bank Conflicts and Padding in the Blackwell Era

16. Distributed Shared Memory in Hopper and Blackwell

17. Cooperative Groups for Heterogeneous Workoads

18. Asynchronous Data Movement with Copy Async

19. Transactional Memory and Atomic Operations at Scale

20. Designing for High Instruction-Level Parallelism

21. Exploiting Thread-Level Parallelism Limits

22. Tensor Cores: 4th and 5th Generation Deep Dive

23. Mixed-Precision Arithmetic: Mastering FP8 and FP4

24. The Transformer Engine: Hardware-Accelerated Attention

25. Implementing Custom Matrix Multiply-Accumulate Kernels

26. CUDA Graphs: Eliminating CPU-side Overhead

27. Stream-Ordered Memory Allocation and Management

28. Multi-Instance GPU for High-Density Computing

29. Confidential Computing on H100 and B200

30. NVLink 5: High-Speed Interconnect Topologies

31. NVSwitch Fabric: Designing Scalable GPU Clusters

32. GPUDirect RDMA and Storage Technologies

33. Collective Communications: NCCL Internal Mechanics

34. Optimizing All-Reduce and All-to-All Operations

35. Pipelining Workloads with CUDA Streams and Events

36. JIT Compilation and Runtime Kernel Generation

37. Link-Time Optimization in CUDA Applications

38. Advanced Debugging with NVIDIA Nsight Systems

39. Profiling Kernels with Nsight Compute: Metrics and Baselines

40. Roofline Analysis for Performance Bottleneck Identification

41. Managing Occupancy and Warp Schedulers

42. High-Performance Sparse Matrix Computations

43. Fast Fourier Transforms at Exascale

44. Graph Analytics and Irregular Parallelism

45. Accelerating Deep Learning Inference with TensorRT

46. Custom Layers and Plugin Development for AI

47. Dynamic Programming and Sequence Alignment on GPUs

48. Monte Carlo Simulations and RNG Optimization

49. Cryptographic Primitives and Blockchain Acceleration

50. CFD and Molecular Dynamics: Real-world HPC Scaling

51. Ray Tracing Cores for Non-Graphics Workloads

52. Video Encoding and Decoding via NVENC and NVDEC

53. Optical Flow and Computer Vision Hardware Acceleration

54. Unified Memory: Advanced Prefetching and Advice

55. Heterogeneous Memory Management on Linux

56. Multi-GPU Load Balancing and Task Distribution

57. Zero-Copy Memory and PCIe Gen 6 Considerations

58. Building High-Performance Data Loaders for AI

59. Quantization Algorithms for Large Language Models

60. Kernel Fusion Strategies for Complex Pipelines

61. Autotuning Kernels for Cross-Architecture Portability

62. The Future of GPGPU: Beyond Blackwell

63. Hardware-Software Co-Design for Specialized Domains

64. Implementing Software-Managed Caches in Shared Memory

65. Bit-Manipulation Instructions and Advanced Integer Math

66. Handling Large Scale Data with NVIDIA Multi-Node Toolkit

67. Virtual Memory Management API Deep Dive

68. Cooperative Launch and Grid-Level Synchronization

69. Reducing Tail Latency in Real-time GPU Systems

70. Optimizing for the NVIDIA Grace Hopper Superchip

71. The Grace CPU and LPDDR5X Memory Integration

72. C++ Standard Parallelism for NVIDIA GPUs

73. Python Performance with CuPy and Numba

74. Integrating CUDA with Rust and Modern Systems Languages

75. Formal Verification of CUDA Kernels

76. Security Vulnerabilities in GPGPU Kernels

77. Energy-Efficient GPU Programming

78. Managing Thermal Throttling and Power Limits

79. CI/CD Pipelines for High-Performance Computing

80. Deploying CUDA at Scale: Kubernetes and Slurm

81. Debugging Race Conditions and Memory Corruption

82. Using Sanitizers for CUDA Memory Validation

83. Implementing Custom Allocators for GPU Memory

84. Exploring the CUDA Driver API vs Runtime API

85. Static Analysis Tools for Parallel Code

86. Performance Portability with Kokkos and RAJA

87. OpenMP Offloading for NVIDIA Hardware

88. The CUTLASS Library: Templates for Matrix Multiplies

89. CuTe: A New Abstraction for Tiled Layouts

90. NVIDIA Warp: A Framework for Physics Simulation

91. Optimizing Stencil Computations and Finite Difference

92. Sort Scan and Reduction: The Building Blocks of Parallelism

93. Designing Low-Latency Audio and Signal Processing Kernels

94. Exploiting Hardware-Based Data Compression

95. Case Study: Optimizing Llama-3 for Blackwell

96. Case Study: Climate Modeling at Petabyte Scales

97. Case Study: Real-time Ray Traced Scientific Visualization

98. Sustaining Performance in Long-Running GPU Services

99. Community Ecosystem and Staying Current with CUDA Research

100. Masterclass Conclusion: The Path to Zero-Overhead Computing

Team Discounts

Team Discounts

Get a team discount on this book!

  • Up to 3 members

    Minimum price
    $47.00
    Suggested price
    $72.00
  • Up to 5 members

    Minimum price
    $76.00
    Suggested price
    $116
  • Up to 10 members

    Minimum price
    $133
    Suggested price
    $203
  • Up to 15 members

    Minimum price
    $190
    Suggested price
    $290
  • Up to 25 members

    Minimum price
    $285
    Suggested price
    $435

Author

About the Author

Krzysztof Rybiński

I am an independent technology developer and systems engineer who built my technical path largely through self-directed engineering, experimentation, and continuous learning outside a traditional academic or corporate technology career.

My professional background began far from the technology industry. I spent years working in manufacturing, while independently developing my knowledge of software engineering, computer systems, and advanced computing. Over time, that self-directed work evolved into a broad technical practice spanning autonomous AI, cybersecurity, systems programming, GPU computing, automation, and advanced computational architectures.

Today, I design, build, and publish projects involving agentic AI, autonomous defense systems, SIEM/EDR integration, secure software architecture, C/C++, Go, Python, CUDA, quantum computing, cryptography, and privacy-oriented local AI infrastructure.

I approach technology from a systems perspective — from low-level software, memory architecture, and GPU performance to distributed systems, intelligent agents, and high-assurance security architectures.

I also explore aerospace and high-assurance software concepts, including safety-critical architectures, multi-level security, cross-domain solutions, and advanced computational systems.

Alongside active development, I publish long-form engineering projects covering AI, cybersecurity, cloud engineering, quantum computing, GPU programming, cryptography, automation, blockchain, and aerospace engineering.

My current focus is on autonomous software agents, privacy-first local infrastructure, advanced computing, and reliable systems designed to operate with a high degree of independence.

I am open to opportunities involving AI engineering, cybersecurity, software engineering, autonomous systems, HPC/GPU computing, and advanced technology development.

https://businessofmachines.blogspot.com/

https://learn.microsoft.com/en-us/users/machinadeusex/

https://g.dev/machinadeusex

https://github.com/porucznikswext-source

https://www.linkedin.com/in/krzysztof-r-93a37b287/

Get the free sample chapters

Click the buttons to get the free sample in PDF or EPUB, or read the sample online here

Also by the Author

Also by the Author

The Leanpub 60 Day 100% Happiness Guarantee

Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.

See full terms...

Earn $8 on a $10 Purchase, and $16 on a $20 Purchase

We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.

(Yes, some authors have already earned much more than that on Leanpub.)

Learn more about writing on Leanpub

Free Updates. DRM Free.

If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).

Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.

Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.

Learn more about Leanpub's ebook formats and where to read them

Write and Publish on Leanpub

You can use Leanpub to easily write, publish and sell in-progress and completed ebooks and online courses!

Leanpub is a powerful platform for serious authors, combining a simple, elegant writing and publishing workflow with a store focused on selling in-progress ebooks.

Leanpub is a magical typewriter for authors: just write in plain text, and to publish your ebook, just click a button. (Or, if you are producing your ebook your own way, you can even upload your own PDF and/or EPUB files and then publish with one click!) It really is that easy.

Learn more about writing on Leanpub