19 in-depth guides · GPU architecture, memory, parallelism, and optimization
CUDA organizes parallel work into a three-level hierarchy: threads → blocks → grids. Every kernel launch defines how many blocks and how many threads per block to use.
A warp is 32 threads that the GPU schedules and executes as a single unit. Understanding warp behavior is essential for reasoning about performance.
A CUDA stream is an ordered queue of GPU operations (kernel launches, memory copies, events) that execute in order. Operations in different streams may overlap.
Cooperative Groups (CUDA 9+) provide a composable, type-safe API for synchronizing and communicating across arbitrary subsets of threads — beyond the fixed `__syncthreads()` model.
Dynamic Parallelism (CUDA 5.0+, CC 3.5+) allows kernels to launch child kernels directly from the device, without returning control to the host. It enables adaptive algorithms where the parallel structure isn't known until runtime.
Modern AI and HPC systems use 2–8 GPUs connected via PCIe or NVLink. CUDA exposes each GPU as a separate device; you must explicitly manage which device is active and how data moves between them.
CUDA exposes several distinct memory spaces with different scope, lifetime, latency, and size. Choosing the right memory for each use case is the single most impactful optimization lever available.
Global memory (device DRAM) is served in 128-byte cache line transactions. Whether a warp of 32 threads consumes 1 transaction or 32 depends entirely on the access pattern.
Shared memory is on-chip SRAM with ~32-cycle latency vs ~800 cycles for global DRAM. It is the primary tool for data reuse within a thread block and for staging non-coalesced global memory accesses.
An atomic operation reads, modifies, and writes a memory location as an indivisible unit — no other thread can observe the location in an intermediate state. CUDA provides hardware-accelerated atomics in global and shared memory.
CUDA provides hardware-accelerated texture memory (read-only, cached, with filtering) and surface memory (read-write image objects). Both are backed by global DRAM but accessed through a dedicated texture/surface cache optimized for 2D spatial locality.
Occupancy is the ratio of active warps to the maximum number of warps an SM can support. It determines how effectively the GPU can hide memory latency.
Tensor Cores are dedicated matrix-multiply-accumulate (MMA) units introduced in Volta (V100). They compute a small matrix product (e.g. 16×16×16) in a single clock cycle, delivering >4× the FLOPs of standard CUDA cores.
Reduction maps an array of N values to a single value (sum, max, count) using an associative operator. The naive sequential approach is O(N); the parallel approach achieves O(N/P + log P) — the dominant pattern in GPU programming.
A prefix scan (prefix sum) produces a running cumulative result: given `[a0, a1, a2, a3]`, the inclusive scan is `[a0, a0+a1, a0+a1+a2, a0+a1+a2+a3]`. It is a fundamental building block for stream compaction, sorting, histogram normalization, and load balancing.
Matrix multiplication `C = A × B` (SGEMM — Single-precision General Matrix Multiply) is the most important compute-bound kernel in GPU programming. It is compute-bound: each element of C requires N multiply-add operations but is written only once.
Writing optimized CUDA kernels from scratch for every standard algorithm is unnecessary — NVIDIA provides two libraries that cover most data-parallel primitives at near-peak performance.
CUDA operations fail silently by default. All CUDA runtime functions return a `cudaError_t` code; all kernel launches are asynchronous and can't return a value. Building robust CUDA code requires systematic error checking.
Profiling is not optional — human intuition about GPU bottlenecks is wrong surprisingly often. NVIDIA provides two complementary tools: Nsight Systems (system-level timeline) and Nsight Compute (kernel-level deep analysis).