Parallelism

GPU Thread Hierarchy

GPU Thread Hierarchy

CUDA organizes parallel work into a three-level hierarchy: threads → blocks → grids. Every kernel launch defines how many blocks and how many threads per block to use.

Threads

The basic unit of execution. Each thread runs the same kernel code but operates on different data identified by its built-in index variables:

  • ▸threadIdx.x / .y / .z — thread's position within its block (3D)
  • ▸blockIdx.x / .y / .z — block's position within the grid (3D)
  • ▸blockDim.x / .y / .z — dimensions of the block (threads per dimension)
  • ▸gridDim.x / .y / .z — dimensions of the grid (blocks per dimension)
  • ▸warpSize — always 32 on all current NVIDIA GPUs

Thread Blocks

Threads are grouped into thread blocks. Threads within a block can:

  • ▸Communicate via shared memory (on-chip SRAM, ~100× faster than global memory)
  • ▸Synchronize with __syncthreads() — a barrier all threads in the block must reach before any proceed

Thread blocks execute independently — in any order, in parallel or serial. This independence is what lets CUDA programs scale from small GPUs to large ones without code changes.

Limits per block:

  • ▸Maximum 1024 threads total (any shape: 256×1×1, 32×32×1, 8×8×16, …)
  • ▸Block dimensions should be multiples of 32 (the warp size) to avoid idle threads in the last warp

Grids

Blocks are organized into a grid. Grids and blocks can be 1D, 2D, or 3D — use dim3 to specify:

c
dim3 threadsPerBlock(16, 16);                  // 256 threads per block, 2D layout
dim3 numBlocks((N+15)/16, (N+15)/16);          // ceil(N/16) × ceil(N/16) blocks
MatAdd<<<numBlocks, threadsPerBlock>>>(A, B, C);

Global Thread Index

For 1D kernels:

i = blockIdx.x * blockDim.x + threadIdx.x

For 2D kernels (e.g., image processing):

x = blockIdx.x * blockDim.x + threadIdx.x // column y = blockIdx.y * blockDim.y + threadIdx.y // row

Thread Block Clusters (Compute Capability 9.0+)

Hopper (H100) introduced a fourth level: clusters. Multiple blocks in a cluster are co-scheduled on the same GPU Processing Cluster (GPC) and can share data via Distributed Shared Memory without going through global memory. Maximum portable cluster size: 8 blocks.

c
// Declare a 2-block cluster at compile time
__global__ void __cluster_dims__(2, 1, 1) clusterKernel(float *in, float *out) {
    // blocks in the same cluster can access each other's shared memory
}

Key Rules

| Rule | Why | |---|---| | Blocks execute independently | Enables automatic scaling across GPU sizes | | Only intra-block sync (__syncthreads) is cheap | Cross-block sync requires global memory + extra kernels | | Block size should be a multiple of 32 | Avoids partially-filled warps with idle threads | | Maximum 1024 threads per block | Hard hardware limit on all current GPUs |

// CODE EXAMPLE

example.cu
// 2D kernel: each thread handles one element of an N×N matrix
__global__ void matAdd(const float *A, const float *B, float *C, int N) {
    int x = blockIdx.x * blockDim.x + threadIdx.x;  // column
    int y = blockIdx.y * blockDim.y + threadIdx.y;  // row
    if (x < N && y < N) {
        int i = y * N + x;
        C[i] = A[i] + B[i];
    }
}

int main() {
    const int N = 1024;
    dim3 block(16, 16);                         // 256 threads/block
    dim3 grid((N + 15) / 16, (N + 15) / 16);   // 64×64 = 4096 blocks
    matAdd<<<grid, block>>>(d_A, d_B, d_C, N);
    cudaDeviceSynchronize();
}

// TRY IT

GPU execution
main.cu
loading editor…