An atomic operation reads, modifies, and writes a memory location as an indivisible unit — no other thread can observe the location in an intermediate state. CUDA provides hardware-accelerated atomics in global and shared memory.
| Function | Operation | Types |
|---|---|---|
| atomicAdd(addr, val) | *addr += val | int, uint, float, double, half2 |
| atomicSub(addr, val) | *addr -= val | int, uint |
| atomicMin/Max(addr, val) | Min or max update | int, uint (float on CC 3.5+) |
| atomicAnd/Or/Xor(addr, val) | Bitwise op | int, uint |
| atomicExch(addr, val) | Swap, return old | int, uint, float |
| atomicCAS(addr, compare, val) | Compare-and-swap | int, uint, ull |
| atomicInc/Dec(addr, max) | Modular increment/decrement | uint |
All return the old value at addr before the operation.
atomicCAS is the primitive from which all others can be built:
// Implement atomicMin for float (no native HW instruction on older GPUs)
__device__ float atomicMinFloat(float *addr, float val) {
int *addr_i = (int *)addr;
int old = *addr_i, assumed;
do {
assumed = old;
float current = __int_as_float(old);
if (val >= current) break;
old = atomicCAS(addr_i, assumed, __float_as_int(val));
} while (assumed != old);
return __int_as_float(old);
}
The loop retries until assumed == old, meaning no other thread modified the location between the read and the CAS.
All atomics to the same address serialize. A histogram with 256 bins and 1M threads → potentially 4000 threads contending per bin.
Mitigation strategies:
__global__ void histogram(const int *data, int *hist, int n) {
__shared__ int s_hist[256];
if (threadIdx.x < 256) s_hist[threadIdx.x] = 0;
__syncthreads();
int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i < n) atomicAdd(&s_hist[data[i]], 1); // fast: shared memory
__syncthreads();
// Flush block-local counts to global with one atomic per bin per block
if (threadIdx.x < 256) atomicAdd(&hist[threadIdx.x], s_hist[threadIdx.x]);
}
__ballot_sync to count how many threads in a warp increment the same bin, then do a single atomic per unique bin.cuda::atomic (CUDA 11+)The CUDA C++ <cuda/atomic> header provides C++20-style atomics with explicit memory orders:
#include <cuda/atomic>
__device__ cuda::atomic<int, cuda::thread_scope_device> counter{0};
__global__ void increment() {
counter.fetch_add(1, cuda::memory_order_relaxed); // no ordering guarantee
}
Available memory orders (weakest to strongest): relaxed, acquire, release, acq_rel, seq_cst.
Use the weakest order that is correct — relaxed for independent counters, acquire/release for producer-consumer patterns.
| Scope | Constant | When to use |
|---|---|---|
| Thread block | cuda::thread_scope_block | Shared memory atomics within a block |
| Device | cuda::thread_scope_device | Global memory atomics within one GPU |
| System | cuda::thread_scope_system | Host-device communication via pinned memory |
// CODE EXAMPLE
// Histogram using shared memory atomics (fast) vs global atomics (slow)
#include <cuda_runtime.h>
#define BINS 256
#define BLOCK 256
// FAST: accumulate per-block in shared memory, one atomic to global per bin
__global__ void histShared(const unsigned char *data, int *hist, int n) {
__shared__ int s[BINS];
if (threadIdx.x < BINS) s[threadIdx.x] = 0;
__syncthreads();
int i = blockIdx.x * BLOCK + threadIdx.x;
if (i < n) atomicAdd(&s[data[i]], 1);
__syncthreads();
if (threadIdx.x < BINS) atomicAdd(&hist[threadIdx.x], s[threadIdx.x]);
}
// Demonstrate atomicCAS-based lock for a critical section
__device__ int g_lock = 0;
__global__ void criticalSection(int *shared_val) {
// Spin-lock: not recommended for production, illustrative only
bool done = false;
while (!done) {
if (atomicCAS(&g_lock, 0, 1) == 0) {
// critical section
(*shared_val)++;
__threadfence(); // ensure write is visible before unlock
atomicExch(&g_lock, 0); // release lock
done = true;
}
}
}// TRY IT
GPU execution// PRACTICE PROJECTS
// ALL CONCEPTS
Browse reference