Memory

CUDA Memory Hierarchy

CUDA Memory Hierarchy

CUDA exposes several distinct memory spaces with different scope, lifetime, latency, and size. Choosing the right memory for each use case is the single most impactful optimization lever available.

Overview

| Memory | Scope | Lifetime | Latency | Size (typical) | |---|---|---|---|---| | Registers | Per-thread | Kernel | ~1 cycle | ~256 KB/SM | | Local memory | Per-thread | Kernel | ~800 cycles | Limited by DRAM | | Shared memory | Per-block | Block | ~32 cycles | 48–96 KB/SM | | L1 / Texture cache | Per-SM | — | ~32 cycles | 32–128 KB/SM | | L2 cache | Device | — | ~200 cycles | 4–40 MB | | Global memory | All threads | Application | ~600–800 cycles | GBs (DRAM) | | Constant memory | All threads (read-only) | Application | ~1 cycle (broadcast) | 64 KB | | Texture memory | All threads (read-only) | Application | ~32 cycles (cached) | Up to DRAM size |

Registers

The fastest storage — compiler assigns them automatically. Each SM has a fixed register file (~65,536 × 32-bit registers on Ampere/Hopper) shared among all concurrently active threads.

Register spilling: when a kernel needs more registers than the hardware limit per thread, the overflow spills to local memory — which lives in global DRAM and has ~800-cycle latency. Avoid spilling:

nvcc --ptxas-options=-v kernel.cu # see: registers per thread, spilled stores/loads nvcc -maxrregcount=32 kernel.cu # hard cap (may cause intentional spilling)

Shared Memory

On-chip SRAM declared with __shared__. Latency ~32 cycles vs ~800 for global memory. Shared among all threads in a block, allocated per-block.

On Ampere, each SM has 192 KB of on-chip SRAM split between L1 cache and shared memory (configurable):

c
cudaFuncSetAttribute(myKernel,
    cudaFuncAttributePreferredSharedMemoryCarveout, 50); // 50% shared, 50% L1

More shared memory per block → fewer concurrent blocks per SM → lower occupancy. Find the right balance.

Global Memory

The main GPU DRAM — accessible from all threads, persists across kernel launches. Managed with cudaMalloc / cudaFree.

Bandwidth is everything: A100 offers ~2 TB/s, H100 HBM3 ~3.35 TB/s. Achieving peak requires coalesced access (consecutive threads accessing consecutive addresses).

Constant Memory

64 KB, read-only, with a dedicated cache. When all threads in a warp read the same address, it broadcasts in a single cycle — ideal for kernel parameters, filter weights, and lookup tables:

c
__constant__ float c_kernel[9];  // declared at file scope
cudaMemcpyToSymbol(c_kernel, h_kernel, 9 * sizeof(float));

Texture Memory

Cached through the texture cache with 2D spatial locality. Good for image processing stencils where you access 2D neighborhoods. Supports hardware bilinear interpolation and boundary clamping.

Unified Memory (cudaMallocManaged)

Single address space accessible from both host and device. CUDA migrates pages automatically on demand:

c
float *data;
cudaMallocManaged(&data, N * sizeof(float));
data[0] = 1.0f;                               // CPU access
myKernel<<<grid, block>>>(data, N);           // GPU access
cudaDeviceSynchronize();
printf("%f\n", data[0]);                      // CPU reads GPU result
cudaFree(data);

Supports oversubscription on Pascal+ GPUs (device can address more memory than physically present via page faulting).

Pinned (Page-Locked) Host Memory

Regular malloc allocates pageable memory. PCIe DMA transfers require pinned memory — if you use malloc, CUDA must internally copy to a pinned staging buffer first (halving effective transfer bandwidth). Use cudaMallocHost for any buffer you transfer repeatedly:

c
float *h_buf;
cudaMallocHost(&h_buf, N * sizeof(float));   // pinned — DMA can access directly
// Required for cudaMemcpyAsync
cudaFreeHost(h_buf);

// CODE EXAMPLE

example.cu
// Using all major memory spaces in one kernel
__constant__ float c_scale;                    // constant memory (file scope)

__global__ void memDemoKernel(
    const float * __restrict__ g_in,           // global memory
    float *g_out, int n)
{
    __shared__ float s_tile[256];              // shared memory

    int tid = threadIdx.x;
    int gid = blockIdx.x * blockDim.x + tid;

    // Coalesced load from global → shared memory
    s_tile[tid] = (gid < n) ? g_in[gid] : 0.f;
    __syncthreads();

    // Compute using a register and constant memory
    float reg = s_tile[tid] * c_scale;         // register × constant

    if (gid < n) g_out[gid] = reg;
}

int main() {
    // Copy to constant memory
    float scale = 2.0f;
    cudaMemcpyToSymbol(c_scale, &scale, sizeof(float));

    // Use pinned host memory for fast transfers
    float *h_data;
    cudaMallocHost(&h_data, N * sizeof(float));
    // ... fill h_data, cudaMemcpy, launch kernel ...
    cudaFreeHost(h_data);
}

// TRY IT

GPU execution
main.cu
loading editor…