Memory

Atomic Operations and Memory Ordering

Atomic Operations and Memory Ordering

An atomic operation reads, modifies, and writes a memory location as an indivisible unit — no other thread can observe the location in an intermediate state. CUDA provides hardware-accelerated atomics in global and shared memory.

Built-in Atomic Functions

| Function | Operation | Types | |---|---|---| | atomicAdd(addr, val) | *addr += val | int, uint, float, double, half2 | | atomicSub(addr, val) | *addr -= val | int, uint | | atomicMin/Max(addr, val) | Min or max update | int, uint (float on CC 3.5+) | | atomicAnd/Or/Xor(addr, val) | Bitwise op | int, uint | | atomicExch(addr, val) | Swap, return old | int, uint, float | | atomicCAS(addr, compare, val) | Compare-and-swap | int, uint, ull | | atomicInc/Dec(addr, max) | Modular increment/decrement | uint |

All return the old value at addr before the operation.

Compare-And-Swap (CAS)

atomicCAS is the primitive from which all others can be built:

c
// Implement atomicMin for float (no native HW instruction on older GPUs)
__device__ float atomicMinFloat(float *addr, float val) {
    int *addr_i = (int *)addr;
    int old = *addr_i, assumed;
    do {
        assumed = old;
        float current = __int_as_float(old);
        if (val >= current) break;
        old = atomicCAS(addr_i, assumed, __float_as_int(val));
    } while (assumed != old);
    return __int_as_float(old);
}

The loop retries until assumed == old, meaning no other thread modified the location between the read and the CAS.

Contention and Performance

All atomics to the same address serialize. A histogram with 256 bins and 1M threads → potentially 4000 threads contending per bin.

Mitigation strategies:

  1. ▸Shared memory atomics + reduction — accumulate per-block in shared memory, flush to global once:
c
__global__ void histogram(const int *data, int *hist, int n) {
    __shared__ int s_hist[256];
    if (threadIdx.x < 256) s_hist[threadIdx.x] = 0;
    __syncthreads();

    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) atomicAdd(&s_hist[data[i]], 1);   // fast: shared memory
    __syncthreads();

    // Flush block-local counts to global with one atomic per bin per block
    if (threadIdx.x < 256) atomicAdd(&hist[threadIdx.x], s_hist[threadIdx.x]);
}
  1. ▸Thread-private partial histograms — each thread accumulates in private arrays, merge at end.
  2. ▸Warp-level aggregation — use __ballot_sync to count how many threads in a warp increment the same bin, then do a single atomic per unique bin.

Memory Ordering: cuda::atomic (CUDA 11+)

The CUDA C++ <cuda/atomic> header provides C++20-style atomics with explicit memory orders:

c
#include <cuda/atomic>

__device__ cuda::atomic<int, cuda::thread_scope_device> counter{0};

__global__ void increment() {
    counter.fetch_add(1, cuda::memory_order_relaxed); // no ordering guarantee
}

Available memory orders (weakest to strongest): relaxed, acquire, release, acq_rel, seq_cst.

Use the weakest order that is correct — relaxed for independent counters, acquire/release for producer-consumer patterns.

Atomic Scope

| Scope | Constant | When to use | |---|---|---| | Thread block | cuda::thread_scope_block | Shared memory atomics within a block | | Device | cuda::thread_scope_device | Global memory atomics within one GPU | | System | cuda::thread_scope_system | Host-device communication via pinned memory |

// CODE EXAMPLE

example.cu
// Histogram using shared memory atomics (fast) vs global atomics (slow)
#include <cuda_runtime.h>
#define BINS 256
#define BLOCK 256

// FAST: accumulate per-block in shared memory, one atomic to global per bin
__global__ void histShared(const unsigned char *data, int *hist, int n) {
    __shared__ int s[BINS];
    if (threadIdx.x < BINS) s[threadIdx.x] = 0;
    __syncthreads();

    int i = blockIdx.x * BLOCK + threadIdx.x;
    if (i < n) atomicAdd(&s[data[i]], 1);
    __syncthreads();

    if (threadIdx.x < BINS) atomicAdd(&hist[threadIdx.x], s[threadIdx.x]);
}

// Demonstrate atomicCAS-based lock for a critical section
__device__ int g_lock = 0;

__global__ void criticalSection(int *shared_val) {
    // Spin-lock: not recommended for production, illustrative only
    bool done = false;
    while (!done) {
        if (atomicCAS(&g_lock, 0, 1) == 0) {
            // critical section
            (*shared_val)++;
            __threadfence();         // ensure write is visible before unlock
            atomicExch(&g_lock, 0); // release lock
            done = true;
        }
    }
}

// TRY IT

GPU execution
main.cu
loading editor…