Tooling

Profiling with Nsight Compute and Nsight Systems

Profiling with Nsight Compute and Nsight Systems

Profiling is not optional — human intuition about GPU bottlenecks is wrong surprisingly often. NVIDIA provides two complementary tools: Nsight Systems (system-level timeline) and Nsight Compute (kernel-level deep analysis).

The Roofline Model

Before profiling, understand whether your kernel is compute-bound or memory-bound:

  • ▸Compute-bound: execution time limited by arithmetic throughput (FLOPs)
  • ▸Memory-bound: execution time limited by bandwidth (bytes/second)

Arithmetic intensity = FLOPs / bytes transferred to/from DRAM

If your kernel's arithmetic intensity is below the ridge point (peak FLOPs / peak bandwidth), it's memory-bound. For A100: 19.5 TFLOPS / 2 TB/s = ~9.75 FLOP/byte ridge point.

Nsight Systems: Timeline View

Use Nsight Systems (nsys) first — it shows the big picture: are kernels overlapping with copies? Is there GPU idle time? Are there many small kernel launches with high CPU overhead?

bash
# Profile the full application
nsys profile --stats=true -o report.nsys-rep ./my_app

# View in GUI
nsys-ui report.nsys-rep

# Or print summary to terminal
nsys stats report.nsys-rep

Key things to look for:

  • ▸Long gaps between kernels (CPU bottleneck or missing streams)
  • ▸Memory copies not overlapping with compute (missing async + pinned memory)
  • ▸Many tiny kernels (consider CUDA Graphs or kernel fusion)

Nsight Compute: Per-Kernel Analysis

ncu drills into a single kernel execution. It measures hardware counters in multiple passes:

bash
# Profile all kernels
ncu --set full -o report ./my_app

# Profile only one kernel
ncu --kernel-name myKernel --launch-count 1 -o report ./my_app

# Open GUI
ncu-ui report.ncu-rep

Key Metrics to Read

| Section | Metric | What it tells you | |---|---|---| | SM Throughput | sm__throughput.avg.pct_of_peak | Overall SM utilization | | Memory Throughput | l1tex__throughput, lts__throughput | L1/L2/DRAM utilization | | Warp Stalls | smsp__warp_issue_stalled_*.pct | Where warps are waiting | | Achieved Occupancy | sm__warps_active.avg.pct_of_peak | Active warps vs. theoretical | | L2 Hit Rate | lts__t_sector_hit_rate | L2 cache effectiveness | | Uncoalesced Accesses | l1tex__data_pipe_lsu_wavefronts_mem_lg_cmd_ld.sum | Non-coalesced load transactions |

CUDA Events for Timing

For microbenchmarking individual kernels in code:

c
cudaEvent_t t0, t1;
cudaEventCreate(&t0);
cudaEventCreate(&t1);

cudaEventRecord(t0);
myKernel<<<grid, block>>>(args);
cudaEventRecord(t1);
cudaEventSynchronize(t1);

float ms;
cudaEventElapsedTime(&ms, t0, t1);
printf("Kernel: %.3f ms\n", ms);

// Compute effective bandwidth (for memory-bound kernels)
float bytes = 2.0f * N * sizeof(float);  // 1 read + 1 write
printf("Bandwidth: %.1f GB/s\n", bytes / ms / 1e6f);

cudaEventDestroy(t0);
cudaEventDestroy(t1);

Profiling-Guided Optimization Loop

  1. ▸Profile with Nsight Systems → find top-time kernels
  2. ▸Profile with Nsight Compute → find the binding limiter (memory vs. compute)
  3. ▸If memory-bound: improve coalescing, add shared memory tiling, reduce data movement
  4. ▸If compute-bound: use tensor cores, reduce redundant ops, improve ILP with register tiling
  5. ▸Re-profile to verify improvement → repeat

// CODE EXAMPLE

example.cu
// Self-contained bandwidth measurement using CUDA events
#include <cuda_runtime.h>
#include <cstdio>

__global__ void copyKernel(const float * __restrict__ src,
                           float * __restrict__ dst, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) dst[i] = src[i];
}

int main() {
    const int N     = 1 << 25;  // 32M floats = 128 MB
    const int ITERS = 20;
    const size_t bytes = N * sizeof(float);

    float *d_src, *d_dst;
    cudaMalloc(&d_src, bytes);
    cudaMalloc(&d_dst, bytes);
    cudaMemset(d_src, 0, bytes);

    cudaEvent_t t0, t1;
    cudaEventCreate(&t0);
    cudaEventCreate(&t1);

    int tpb = 256, blocks = (N + tpb - 1) / tpb;

    // Warm-up (fills caches, JIT-compiles)
    copyKernel<<<blocks, tpb>>>(d_src, d_dst, N);
    cudaDeviceSynchronize();

    cudaEventRecord(t0);
    for (int i = 0; i < ITERS; i++)
        copyKernel<<<blocks, tpb>>>(d_src, d_dst, N);
    cudaEventRecord(t1);
    cudaEventSynchronize(t1);

    float ms;
    cudaEventElapsedTime(&ms, t0, t1);
    float avg_ms = ms / ITERS;

    // 2× N bytes: one read + one write
    float bw = 2.0f * bytes / avg_ms / 1e6f;
    printf("Average: %.3f ms  Bandwidth: %.1f GB/s\n", avg_ms, bw);

    cudaFree(d_src); cudaFree(d_dst);
    cudaEventDestroy(t0); cudaEventDestroy(t1);
}

// TRY IT

GPU execution
main.cu
loading editor…