CUDA organizes parallel work into a three-level hierarchy: threads → blocks → grids. Every kernel launch defines how many blocks and how many threads per block to use.
The basic unit of execution. Each thread runs the same kernel code but operates on different data identified by its built-in index variables:
threadIdx.x / .y / .z — thread's position within its block (3D)blockIdx.x / .y / .z — block's position within the grid (3D)blockDim.x / .y / .z — dimensions of the block (threads per dimension)gridDim.x / .y / .z — dimensions of the grid (blocks per dimension)warpSize — always 32 on all current NVIDIA GPUsThreads are grouped into thread blocks. Threads within a block can:
__syncthreads() — a barrier all threads in the block must reach before any proceedThread blocks execute independently — in any order, in parallel or serial. This independence is what lets CUDA programs scale from small GPUs to large ones without code changes.
Limits per block:
Blocks are organized into a grid. Grids and blocks can be 1D, 2D, or 3D — use dim3 to specify:
dim3 threadsPerBlock(16, 16); // 256 threads per block, 2D layout
dim3 numBlocks((N+15)/16, (N+15)/16); // ceil(N/16) × ceil(N/16) blocks
MatAdd<<<numBlocks, threadsPerBlock>>>(A, B, C);
For 1D kernels:
i = blockIdx.x * blockDim.x + threadIdx.x
For 2D kernels (e.g., image processing):
x = blockIdx.x * blockDim.x + threadIdx.x // column
y = blockIdx.y * blockDim.y + threadIdx.y // row
Hopper (H100) introduced a fourth level: clusters. Multiple blocks in a cluster are co-scheduled on the same GPU Processing Cluster (GPC) and can share data via Distributed Shared Memory without going through global memory. Maximum portable cluster size: 8 blocks.
// Declare a 2-block cluster at compile time
__global__ void __cluster_dims__(2, 1, 1) clusterKernel(float *in, float *out) {
// blocks in the same cluster can access each other's shared memory
}
| Rule | Why |
|---|---|
| Blocks execute independently | Enables automatic scaling across GPU sizes |
| Only intra-block sync (__syncthreads) is cheap | Cross-block sync requires global memory + extra kernels |
| Block size should be a multiple of 32 | Avoids partially-filled warps with idle threads |
| Maximum 1024 threads per block | Hard hardware limit on all current GPUs |
// CODE EXAMPLE
// 2D kernel: each thread handles one element of an N×N matrix
__global__ void matAdd(const float *A, const float *B, float *C, int N) {
int x = blockIdx.x * blockDim.x + threadIdx.x; // column
int y = blockIdx.y * blockDim.y + threadIdx.y; // row
if (x < N && y < N) {
int i = y * N + x;
C[i] = A[i] + B[i];
}
}
int main() {
const int N = 1024;
dim3 block(16, 16); // 256 threads/block
dim3 grid((N + 15) / 16, (N + 15) / 16); // 64×64 = 4096 blocks
matAdd<<<grid, block>>>(d_A, d_B, d_C, N);
cudaDeviceSynchronize();
}// TRY IT
GPU execution// PRACTICE PROJECTS
// ALL CONCEPTS
Browse reference