CUDA exposes several distinct memory spaces with different scope, lifetime, latency, and size. Choosing the right memory for each use case is the single most impactful optimization lever available.
| Memory | Scope | Lifetime | Latency | Size (typical) | |---|---|---|---|---| | Registers | Per-thread | Kernel | ~1 cycle | ~256 KB/SM | | Local memory | Per-thread | Kernel | ~800 cycles | Limited by DRAM | | Shared memory | Per-block | Block | ~32 cycles | 48–96 KB/SM | | L1 / Texture cache | Per-SM | — | ~32 cycles | 32–128 KB/SM | | L2 cache | Device | — | ~200 cycles | 4–40 MB | | Global memory | All threads | Application | ~600–800 cycles | GBs (DRAM) | | Constant memory | All threads (read-only) | Application | ~1 cycle (broadcast) | 64 KB | | Texture memory | All threads (read-only) | Application | ~32 cycles (cached) | Up to DRAM size |
The fastest storage — compiler assigns them automatically. Each SM has a fixed register file (~65,536 × 32-bit registers on Ampere/Hopper) shared among all concurrently active threads.
Register spilling: when a kernel needs more registers than the hardware limit per thread, the overflow spills to local memory — which lives in global DRAM and has ~800-cycle latency. Avoid spilling:
nvcc --ptxas-options=-v kernel.cu # see: registers per thread, spilled stores/loads
nvcc -maxrregcount=32 kernel.cu # hard cap (may cause intentional spilling)
On-chip SRAM declared with __shared__. Latency ~32 cycles vs ~800 for global memory. Shared among all threads in a block, allocated per-block.
On Ampere, each SM has 192 KB of on-chip SRAM split between L1 cache and shared memory (configurable):
cudaFuncSetAttribute(myKernel,
cudaFuncAttributePreferredSharedMemoryCarveout, 50); // 50% shared, 50% L1
More shared memory per block → fewer concurrent blocks per SM → lower occupancy. Find the right balance.
The main GPU DRAM — accessible from all threads, persists across kernel launches. Managed with cudaMalloc / cudaFree.
Bandwidth is everything: A100 offers ~2 TB/s, H100 HBM3 ~3.35 TB/s. Achieving peak requires coalesced access (consecutive threads accessing consecutive addresses).
64 KB, read-only, with a dedicated cache. When all threads in a warp read the same address, it broadcasts in a single cycle — ideal for kernel parameters, filter weights, and lookup tables:
__constant__ float c_kernel[9]; // declared at file scope
cudaMemcpyToSymbol(c_kernel, h_kernel, 9 * sizeof(float));
Cached through the texture cache with 2D spatial locality. Good for image processing stencils where you access 2D neighborhoods. Supports hardware bilinear interpolation and boundary clamping.
cudaMallocManaged)Single address space accessible from both host and device. CUDA migrates pages automatically on demand:
float *data;
cudaMallocManaged(&data, N * sizeof(float));
data[0] = 1.0f; // CPU access
myKernel<<<grid, block>>>(data, N); // GPU access
cudaDeviceSynchronize();
printf("%f\n", data[0]); // CPU reads GPU result
cudaFree(data);
Supports oversubscription on Pascal+ GPUs (device can address more memory than physically present via page faulting).
Regular malloc allocates pageable memory. PCIe DMA transfers require pinned memory — if you use malloc, CUDA must internally copy to a pinned staging buffer first (halving effective transfer bandwidth). Use cudaMallocHost for any buffer you transfer repeatedly:
float *h_buf;
cudaMallocHost(&h_buf, N * sizeof(float)); // pinned — DMA can access directly
// Required for cudaMemcpyAsync
cudaFreeHost(h_buf);
// CODE EXAMPLE
// Using all major memory spaces in one kernel
__constant__ float c_scale; // constant memory (file scope)
__global__ void memDemoKernel(
const float * __restrict__ g_in, // global memory
float *g_out, int n)
{
__shared__ float s_tile[256]; // shared memory
int tid = threadIdx.x;
int gid = blockIdx.x * blockDim.x + tid;
// Coalesced load from global → shared memory
s_tile[tid] = (gid < n) ? g_in[gid] : 0.f;
__syncthreads();
// Compute using a register and constant memory
float reg = s_tile[tid] * c_scale; // register × constant
if (gid < n) g_out[gid] = reg;
}
int main() {
// Copy to constant memory
float scale = 2.0f;
cudaMemcpyToSymbol(c_scale, &scale, sizeof(float));
// Use pinned host memory for fast transfers
float *h_data;
cudaMallocHost(&h_data, N * sizeof(float));
// ... fill h_data, cudaMemcpy, launch kernel ...
cudaFreeHost(h_data);
}// TRY IT
GPU execution