Profiling is not optional — human intuition about GPU bottlenecks is wrong surprisingly often. NVIDIA provides two complementary tools: Nsight Systems (system-level timeline) and Nsight Compute (kernel-level deep analysis).
Before profiling, understand whether your kernel is compute-bound or memory-bound:
Arithmetic intensity = FLOPs / bytes transferred to/from DRAM
If your kernel's arithmetic intensity is below the ridge point (peak FLOPs / peak bandwidth), it's memory-bound. For A100: 19.5 TFLOPS / 2 TB/s = ~9.75 FLOP/byte ridge point.
Use Nsight Systems (nsys) first — it shows the big picture: are kernels overlapping with copies? Is there GPU idle time? Are there many small kernel launches with high CPU overhead?
# Profile the full application
nsys profile --stats=true -o report.nsys-rep ./my_app
# View in GUI
nsys-ui report.nsys-rep
# Or print summary to terminal
nsys stats report.nsys-rep
Key things to look for:
ncu drills into a single kernel execution. It measures hardware counters in multiple passes:
# Profile all kernels
ncu --set full -o report ./my_app
# Profile only one kernel
ncu --kernel-name myKernel --launch-count 1 -o report ./my_app
# Open GUI
ncu-ui report.ncu-rep
| Section | Metric | What it tells you | |---|---|---| | SM Throughput | sm__throughput.avg.pct_of_peak | Overall SM utilization | | Memory Throughput | l1tex__throughput, lts__throughput | L1/L2/DRAM utilization | | Warp Stalls | smsp__warp_issue_stalled_*.pct | Where warps are waiting | | Achieved Occupancy | sm__warps_active.avg.pct_of_peak | Active warps vs. theoretical | | L2 Hit Rate | lts__t_sector_hit_rate | L2 cache effectiveness | | Uncoalesced Accesses | l1tex__data_pipe_lsu_wavefronts_mem_lg_cmd_ld.sum | Non-coalesced load transactions |
For microbenchmarking individual kernels in code:
cudaEvent_t t0, t1;
cudaEventCreate(&t0);
cudaEventCreate(&t1);
cudaEventRecord(t0);
myKernel<<<grid, block>>>(args);
cudaEventRecord(t1);
cudaEventSynchronize(t1);
float ms;
cudaEventElapsedTime(&ms, t0, t1);
printf("Kernel: %.3f ms\n", ms);
// Compute effective bandwidth (for memory-bound kernels)
float bytes = 2.0f * N * sizeof(float); // 1 read + 1 write
printf("Bandwidth: %.1f GB/s\n", bytes / ms / 1e6f);
cudaEventDestroy(t0);
cudaEventDestroy(t1);
// CODE EXAMPLE
// Self-contained bandwidth measurement using CUDA events
#include <cuda_runtime.h>
#include <cstdio>
__global__ void copyKernel(const float * __restrict__ src,
float * __restrict__ dst, int n) {
int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i < n) dst[i] = src[i];
}
int main() {
const int N = 1 << 25; // 32M floats = 128 MB
const int ITERS = 20;
const size_t bytes = N * sizeof(float);
float *d_src, *d_dst;
cudaMalloc(&d_src, bytes);
cudaMalloc(&d_dst, bytes);
cudaMemset(d_src, 0, bytes);
cudaEvent_t t0, t1;
cudaEventCreate(&t0);
cudaEventCreate(&t1);
int tpb = 256, blocks = (N + tpb - 1) / tpb;
// Warm-up (fills caches, JIT-compiles)
copyKernel<<<blocks, tpb>>>(d_src, d_dst, N);
cudaDeviceSynchronize();
cudaEventRecord(t0);
for (int i = 0; i < ITERS; i++)
copyKernel<<<blocks, tpb>>>(d_src, d_dst, N);
cudaEventRecord(t1);
cudaEventSynchronize(t1);
float ms;
cudaEventElapsedTime(&ms, t0, t1);
float avg_ms = ms / ITERS;
// 2× N bytes: one read + one write
float bw = 2.0f * bytes / avg_ms / 1e6f;
printf("Average: %.3f ms Bandwidth: %.1f GB/s\n", avg_ms, bw);
cudaFree(d_src); cudaFree(d_dst);
cudaEventDestroy(t0); cudaEventDestroy(t1);
}// TRY IT
GPU execution