Optimization

Tensor Cores and the WMMA API

Tensor Cores and the WMMA API

Tensor Cores are dedicated matrix-multiply-accumulate (MMA) units introduced in Volta (V100). They compute a small matrix product (e.g. 16×16×16) in a single clock cycle, delivering >4× the FLOPs of standard CUDA cores.

Why Tensor Cores Exist

A dense GEMM is dominated by fused multiply-add (FMA) operations. Regular CUDA cores execute one FMA per thread per clock. Tensor Cores execute a 16×16×16 FP16 matrix multiply per warp per clock — 256 FMAs in one instruction vs. 32 from regular cores.

| GPU | FP32 (CUDA cores) | FP16 Tensor (WMMA) | |---|---|---| | V100 | 14 TFLOPS | 112 TFLOPS | | A100 | 19.5 TFLOPS | 312 TFLOPS (TF32), 624 TFLOPS (FP16) | | H100 | 67 TFLOPS | 1979 TFLOPS (FP8) |

WMMA: Warp Matrix Multiply-Accumulate

The nvcuda::wmma API exposes tensor cores at the warp level. All 32 threads in a warp cooperate to hold a fragment (a distributed register tile):

c
#include <mma.h>
using namespace nvcuda::wmma;

// Fragment types: matrix_a, matrix_b, accumulator
fragment<matrix_a, 16, 16, 16, half, row_major> a_frag;
fragment<matrix_b, 16, 16, 16, half, col_major> b_frag;
fragment<accumulator, 16, 16, 16, float>         c_frag;

// Initialize accumulator to zero
fill_fragment(c_frag, 0.0f);

// Load a 16×16 tile of A and B from shared or global memory
load_matrix_sync(a_frag, tile_A_ptr, lda);   // lda = leading dimension
load_matrix_sync(b_frag, tile_B_ptr, ldb);

// Execute: c_frag += a_frag * b_frag (tensor core instruction)
mma_sync(c_frag, a_frag, b_frag, c_frag);

// Store the result tile back to memory
store_matrix_sync(c_ptr, c_frag, ldc, mem_row_major);

Supported Data Types

| Data type pair | Accumulator | Min. Compute Capability | |---|---|---| | FP16 × FP16 | FP16 or FP32 | CC 7.0 (Volta) | | BF16 × BF16 | FP32 | CC 8.0 (Ampere) | | TF32 × TF32 | FP32 | CC 8.0 (Ampere) | | FP8 × FP8 | FP16/FP32 | CC 9.0 (Hopper) | | INT8 × INT8 | INT32 | CC 7.2 (Volta) |

TF32: Drop-in Acceleration

Ampere added TF32 (19-bit mantissa truncated from FP32), enabling FP32 → tensor core automatically:

c
// Enable TF32 for cuBLAS (transparent, ~10× vs FP32 CUDA cores)
cublasSetMathMode(handle, CUBLAS_TF32_TENSOR_OP_MATH);

No code changes required — cuBLAS handles the conversion internally.

Alignment Requirements

WMMA requires:

  • ▸128-byte aligned shared memory pointers
  • ▸Leading dimensions divisible by the tile size (16 for standard WMMA)
  • ▸Input matrices in FP16 for basic WMMA (FP32 inputs need conversion)
c
// Convert FP32 → FP16 on device before WMMA
__global__ void fp32ToFp16(const float *in, half *out, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) out[i] = __float2half(in[i]);
}

Practical Advice

For production: always use cuBLAS or cuDNN — they use WMMA internally with architecture-specific tuning, achieving 95%+ of peak. Write WMMA directly only for custom fused operations (e.g., fused attention) or as a learning exercise.

// CODE EXAMPLE

example.cu
// Minimal WMMA GEMM kernel: C (FP32) = A (FP16) * B (FP16)
#include <mma.h>
#include <cuda_fp16.h>
using namespace nvcuda::wmma;

#define WMMA_M 16
#define WMMA_N 16
#define WMMA_K 16

__global__ void wmmaGemm(
    const half *A, const half *B, float *C, int M, int N, int K)
{
    // Warp-level tile coordinates
    int warpM = (blockIdx.x * blockDim.x + threadIdx.x) / 32;
    int warpN =  blockIdx.y * blockDim.y + threadIdx.y;

    fragment<matrix_a, WMMA_M, WMMA_N, WMMA_K, half, row_major> aFrag;
    fragment<matrix_b, WMMA_M, WMMA_N, WMMA_K, half, col_major> bFrag;
    fragment<accumulator, WMMA_M, WMMA_N, WMMA_K, float>         cFrag;
    fill_fragment(cFrag, 0.0f);

    // Sweep over K tiles
    for (int k = 0; k < K; k += WMMA_K) {
        if (warpM * WMMA_M < M && k < K)
            load_matrix_sync(aFrag, A + warpM * WMMA_M * K + k, K);
        if (warpN * WMMA_N < N && k < K)
            load_matrix_sync(bFrag, B + k * N + warpN * WMMA_N, N);
        mma_sync(cFrag, aFrag, bFrag, cFrag);
    }

    // Store result
    if (warpM * WMMA_M < M && warpN * WMMA_N < N)
        store_matrix_sync(C + warpM * WMMA_M * N + warpN * WMMA_N,
                          cFrag, N, mem_row_major);
}

// TRY IT

GPU execution
main.cu
loading editor…