Parallelism

Cooperative Groups and Warp Primitives

Cooperative Groups and Warp Primitives

Cooperative Groups (CUDA 9+) provide a composable, type-safe API for synchronizing and communicating across arbitrary subsets of threads — beyond the fixed __syncthreads() model.

Thread Group Types

| Group | API | Scope | |---|---|---| | Thread block | cg::this_thread_block() | All threads in current block | | Warp tile (32) | cg::tiled_partition<32>(block) | One full warp | | Sub-warp tile | cg::tiled_partition<N>(block) | N threads (N = 2, 4, 8, 16, or 32) | | Grid group | cg::this_grid() | All threads (cooperative launch required) | | Coalesced group | cg::coalesced_threads() | All currently active threads in warp |

c
#include <cooperative_groups.h>
namespace cg = cooperative_groups;

__global__ void example() {
    cg::thread_block block = cg::this_thread_block();
    block.sync();   // same as __syncthreads()

    // Partition into 32-thread warps
    auto warp = cg::tiled_partition<32>(block);
    warp.sync();    // synchronize this warp only

    // Sub-warp: 4-thread tiles for finer-grained cooperation
    auto quad = cg::tiled_partition<4>(block);
    quad.sync();
}

Group-Level Reduce

Cooperative Groups provides built-in reduce operations that use the fastest available mechanism (shuffles for warps, shared memory for blocks):

c
#include <cooperative_groups/reduce.h>

auto warp = cg::tiled_partition<32>(cg::this_thread_block());
float sum = cg::reduce(warp, val, cg::plus<float>());    // warp sum
float mx  = cg::reduce(warp, val, cg::greater<float>()); // warp max

Warp Shuffle Intrinsics

Exchange register values between threads in a warp — no shared memory, no synchronization needed:

c
// Broadcast: all threads receive the value held by lane srcLane
float v = __shfl_sync(0xffffffff, val, 0);           // broadcast from lane 0

// Shift down: thread i receives the value of thread i+delta
float next = __shfl_down_sync(0xffffffff, val, 1);   // useful for scan

// Butterfly XOR: pairs (0,1), (2,3), ... (used in reductions)
float pair = __shfl_xor_sync(0xffffffff, val, 1);

The first argument is a 32-bit active mask. Use 0xffffffff for full warp participation. All threads named in the mask must execute the instruction.

Warp Vote Functions

Test predicates across all lanes:

c
// 32-bit mask: bit i = 1 if thread i's predicate is true
unsigned mask = __ballot_sync(0xffffffff, val > 0.f);
int count = __popc(mask);   // population count = number of true threads

bool any_true = __any_sync(0xffffffff, val > 0.f);
bool all_true = __all_sync(0xffffffff, val > 0.f);

Grid-Level Cooperation (Cooperative Kernels)

For synchronization across all blocks without ending the kernel, use a cooperative launch. All blocks must be resident simultaneously — the launch size is limited:

c
__global__ void cooperativeKernel(float *data, int n) {
    cg::grid_group grid = cg::this_grid();

    // Phase 1: independent work per block
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) data[i] *= 2.0f;

    grid.sync();  // wait for all blocks to finish phase 1

    // Phase 2: depends on phase 1 results from other blocks
    if (i < n && i > 0) data[i] += data[i - 1];
}

// Must use cudaLaunchCooperativeKernel
void *args[] = { &d_data, &n };
cudaLaunchCooperativeKernel((void*)cooperativeKernel, grid, block, args);

Constraint: grid size ≤ SM count × max concurrent blocks per SM (entire grid must be resident).

// CODE EXAMPLE

example.cu
// Warp reduction using Cooperative Groups reduce API
#include <cooperative_groups.h>
#include <cooperative_groups/reduce.h>
namespace cg = cooperative_groups;

__global__ void cgBlockSum(const float *in, float *out, int n) {
    cg::thread_block      block = cg::this_thread_block();
    cg::thread_block_tile<32> warp = cg::tiled_partition<32>(block);

    __shared__ float warpResults[32];  // one slot per warp

    int gid    = blockIdx.x * blockDim.x + threadIdx.x;
    int lane   = warp.thread_rank();   // 0-31
    int warpId = threadIdx.x / 32;

    float val = (gid < n) ? in[gid] : 0.f;

    // Phase 1: reduce within warp using shuffle (cg::reduce uses __shfl internally)
    float warpSum = cg::reduce(warp, val, cg::plus<float>());

    // Phase 2: lane 0 of each warp writes to shared memory
    if (lane == 0) warpResults[warpId] = warpSum;
    block.sync();   // __syncthreads() via cooperative groups

    // Phase 3: first warp reduces all warp results
    if (warpId == 0) {
        int numWarps = (blockDim.x + 31) / 32;
        float partial = (lane < numWarps) ? warpResults[lane] : 0.f;
        float blockTotal = cg::reduce(warp, partial, cg::plus<float>());
        if (lane == 0) atomicAdd(out, blockTotal);
    }
}

// TRY IT

GPU execution
main.cu
loading editor…