Optimization

Occupancy and Latency Hiding

Occupancy and Latency Hiding

Occupancy is the ratio of active warps to the maximum number of warps an SM can support. It determines how effectively the GPU can hide memory latency.

Why Occupancy Matters

Global memory accesses take ~800 cycles. During those cycles the warp stalls; the warp scheduler switches to another ready warp at zero cost (all state lives in the register file). The more active warps, the more latency can be hidden.

Rule of thumb: 4–8+ active warps per SM are needed to effectively hide global memory latency. On Ampere, the max is 48 warps per SM (1,536 threads).

Three Resource Limits

Whichever resource runs out first determines maximum occupancy:

1. Registers per Thread

Each SM has ~65,536 × 32-bit registers shared across all active threads. If a kernel uses 64 registers/thread with 256 threads/block:

registers per block = 64 × 256 = 16,384 max blocks per SM = floor(65,536 / 16,384) = 4 active warps = 4 × (256/32) = 32 occupancy = 32 / 48 = 67%

Check register usage:

nvcc --ptxas-options=-v kernel.cu # → registers per thread = N, spills = M

2. Shared Memory per Block

If each SM has 96 KB and a block uses 32 KB, only 3 blocks fit (regardless of thread count).

3. Hardware Block/Thread Limits

  • ▸Max 32 blocks per SM (Ampere)
  • ▸Max 48 warps (1,536 threads) per SM (Ampere)
  • ▸Max 1,024 threads per block

CUDA Occupancy API

c
int blockSize, minGridSize;

// Find the block size that maximizes occupancy for your kernel
cudaOccupancyMaxPotentialBlockSize(
    &minGridSize, &blockSize,
    myKernel,
    0,   // dynamic shared memory per block (bytes)
    0    // max block size hint (0 = no limit)
);

// Query how many blocks actually fit per SM
int activeBlocks;
cudaOccupancyMaxActiveBlocksPerMultiprocessor(
    &activeBlocks, myKernel, blockSize, 0);

cudaDeviceProp prop;
cudaGetDeviceProperties(&prop, 0);
float occ = (float)(activeBlocks * blockSize)
          / prop.maxThreadsPerMultiProcessor;
printf("Occupancy: %.1f%%\n", occ * 100);

Occupancy vs. Performance

Higher occupancy ≠ higher performance. A kernel at 50% occupancy that does lots of compute per memory access can outperform a 100% occupancy kernel.

The goal is enough warps to hide latency, not maximizing occupancy:

  • ▸Memory-bound: increasing occupancy helps (more warps = better latency hiding)
  • ▸Compute-bound: occupancy doesn't help; reduce instruction count instead

Profile with Nsight Compute to distinguish these cases.

__launch_bounds__ Hint

Tell the compiler the maximum block size and desired minimum blocks per SM. The compiler uses this to allocate registers more aggressively:

c
// At most 256 threads/block, target at least 4 blocks/SM
__global__ void __launch_bounds__(256, 4) myKernel(float *data, int n) { ... }

// CODE EXAMPLE

example.cu
#include <cuda_runtime.h>
#include <stdio.h>

// __launch_bounds__ guides the compiler's register allocation
__global__ void __launch_bounds__(256, 4) scaleKernel(
    float * __restrict__ out,
    const float * __restrict__ in,
    float scale, int n)
{
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) out[i] = in[i] * scale;
}

int main() {
    // Let CUDA find the optimal block size for occupancy
    int blockSize, minGridSize;
    cudaOccupancyMaxPotentialBlockSize(
        &minGridSize, &blockSize, scaleKernel, 0, 0);

    const int N = 1 << 24;
    int gridSize = (N + blockSize - 1) / blockSize;
    printf("Grid: %d blocks × %d threads\n", gridSize, blockSize);

    // Measure theoretical occupancy
    int activeBlocks;
    cudaOccupancyMaxActiveBlocksPerMultiprocessor(
        &activeBlocks, scaleKernel, blockSize, 0);

    cudaDeviceProp prop;
    cudaGetDeviceProperties(&prop, 0);
    float occ = (float)(activeBlocks * blockSize)
              / prop.maxThreadsPerMultiProcessor;
    printf("Occupancy: %.1f%%\n", occ * 100);

    scaleKernel<<<gridSize, blockSize>>>(d_out, d_in, 2.0f, N);
    cudaDeviceSynchronize();
}

// TRY IT

GPU execution
main.cu
loading editor…