Memory

Shared Memory and Bank Conflicts

Shared Memory and Bank Conflicts

Shared memory is on-chip SRAM with ~32-cycle latency vs ~800 cycles for global DRAM. It is the primary tool for data reuse within a thread block and for staging non-coalesced global memory accesses.

Declaring Shared Memory

Static (size known at compile time):

c
__shared__ float tile[32][32];

Dynamic (size passed at launch):

c
extern __shared__ float s[];
myKernel<<<grid, block, sharedBytes>>>(args);

Memory Banks

Shared memory is physically organized as 32 banks, each 4 bytes wide. Successive 4-byte words map to successive banks cyclically:

bank(address) = (address / 4) % 32

Conflict Rules

| Access pattern | Result | |---|---| | 32 threads → 32 different banks (any permutation) | No conflict — 1 cycle | | Multiple threads → same address | Broadcast — no conflict, 1 cycle | | N threads → different addresses in same bank | N-way conflict — N cycles (serialized) |

Common Bank Conflict Patterns

Stride-1 — no conflict:

c
tile[threadIdx.x]        // thread i → bank i → all 32 banks, no conflict

Stride-2 — 2-way conflict:

c
tile[threadIdx.x * 2]    // thread 0 → bank 0, thread 16 → bank 0 (conflict)

Stride-32 — 32-way conflict:

c
tile[threadIdx.x * 32]   // all 32 threads → bank 0 at different addresses

The Padding Fix

Matrix transpose produces 32-way conflicts when reading tile[threadIdx.x][threadIdx.y]: all 32 elements land in the same bank (one per column).

Fix: pad the inner dimension by 1:

c
__shared__ float tile[32][32 + 1];   // +1 offsets each row's bank assignments

With padding, row r starts at byte offset r * 33 * 4. Element [r][c] maps to bank (r * 33 + c) % 32 — no two rows' column c align to the same bank.

Cost: 32 × 1 × 4 = 128 bytes of wasted memory per tile — negligible.

Shared Memory Capacity

On Ampere (A100), each SM has 192 KB of on-chip SRAM, configurable as shared memory / L1 cache. Default split: 100 KB shared / 28 KB L1 (varies by architecture).

You can request more shared memory per block:

c
cudaFuncSetAttribute(myKernel,
    cudaFuncAttributeMaxDynamicSharedMemorySize, 98304);  // 96 KB

More shared memory per block reduces how many blocks fit simultaneously on an SM (occupancy tradeoff).

When to Use Shared Memory

Use it when:

  • ▸The same data is read multiple times within a block (amortize global memory latency)
  • ▸Global memory access is non-coalesced (stage the data in shared memory with coalesced loads, then access arbitrarily)
  • ▸Implementing reductions, scans, or other intra-block communication

Skip it when:

  • ▸Data is accessed exactly once (L2 cache already helps here)
  • ▸The algorithm naturally has coalesced global memory access with no reuse

// CODE EXAMPLE

example.cu
// Bank-conflict-free matrix transpose: +1 padding eliminates 32-way conflicts
#define TILE 32

__global__ void transposeNoBankConflict(
    const float * __restrict__ in,
    float * __restrict__ out, int N)
{
    // +1 padding shifts each row's bank offset, breaking the 32-way alignment
    __shared__ float tile[TILE][TILE + 1];

    int x = blockIdx.x * TILE + threadIdx.x;
    int y = blockIdx.y * TILE + threadIdx.y;

    // Coalesced read: consecutive threads read consecutive x values
    if (x < N && y < N)
        tile[threadIdx.y][threadIdx.x] = in[y * N + x];
    __syncthreads();

    // Transposed output coordinates
    int ox = blockIdx.y * TILE + threadIdx.x;
    int oy = blockIdx.x * TILE + threadIdx.y;

    // Coalesced write; tile access tile[threadIdx.x][threadIdx.y]
    // is conflict-free due to the +1 padding
    if (ox < N && oy < N)
        out[oy * N + ox] = tile[threadIdx.x][threadIdx.y];
}

// TRY IT

GPU execution
main.cu
loading editor…