NVIDIA/nsight-perf-sdk-for-ai
NVIDIA/nsight-perf-sdk-for-ai has 112 GitHub stars and 5 forks. GitHot measured +1 stars in 24 hours. Last checked 2026-11-15 15:57 UTC. Last push 2026-11-04.
Measured star growth
- in 24h
- +1
- in 7 days
- Update delayed
- in 15 days
- Update delayed
- in 1 month
- Update delayed
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
Always-on, low-overhead GPU and CUDA performance tracing SDK for AI workloads. C++ client APIs, CLI tools, open data model, Perfetto export.
Developer ToolsREADME.md
OriginalNVIDIA Nsight Perf SDK for AI
Version: Beta Preview
The NVIDIA Nsight Perf SDK provides APIs and tools for continuously collecting and consuming performance data from NVIDIA GPUs and the CUDA API. It supports always-on profiling so applications can identify performance issues during execution or by post-processing captured data.
Features
Continuous, low-overhead profiling
NVIDIA Nsight Perf runs continuously in the background. Representative measurements observed <1% overhead for GPU-limited workloads while tracing CUDA driver API calls. Actual overhead depends on the workload, enabled providers, and configuration; see the CUDA benchmarks.
Rich data sources
The SDK can collect:
- CUDA API calls with CPU timestamps, kernel names, and launch dimensions
- GPU workloads, including immediate launches, CUDA graphs, and memory operations
- GPU performance-monitoring counters
- GPU power consumption; see GPU power metrics for hardware, driver, and configuration requirements
- GPU context-switch activity
- NVTX marks and push-pop ranges
- host, operating-system, memory, driver, and GPU information
Configuration and export
The service uses YAML configuration for resource limits, provider selection, GPU metrics, and sampling rates. Trace Stream Databases can be exported with the Perfetto trace exporter and viewed in Perfetto. See the daemon and tools documentation.
Components
nvtrace-clientcontains the target and offline client interfaces and the versioned data formats they consume.nvtrace-toolscontains command-line tools for collecting, inspecting, and exporting traces.nvtrace-samples-cudacontains CUDA samples, tests, and benchmarks.nsight-perf-daemondescribes the NVIDIA Nsight Perf service used by the client and tools.third_partycontains the pinned third-party dependencies used by the build.
Prerequisites
- GPU: NVIDIA Blackwell and Rubin architectures
- CPU:
amd64oraarch64 - OS:
- Linux: Ubuntu 24.04 on
amd64, or a compatible derivative such as DGX OS 7.5 onaarch64
- Linux: Ubuntu 24.04 on
- CUDA Toolkit 13.5 or newer
- NVIDIA driver:
- Linux: 620.00.00 or newer
- Linux kernel-mode drivers starting at 580.95.05 can use the CUDA 13.5 user-mode driver with the CUDA Compatibility package. GPU GR-context tracing has an additional driver-matching requirement.
- Permission to access performance counters when all metrics are required; see Resolving ERR_NVGPUCTRPERM
If a system has both supported and unsupported GPUs, use CUDA_VISIBLE_DEVICES to expose only supported devices.
The source build additionally requires CMake 3.24 or newer and a C++17-capable compiler.
For NVTX tracing, applications built against NVTX 3.7 or newer discover the CUDA driver automatically. Applications built against older NVTX versions require the NVTX CUDA driver injection proxy, selected through NVTX_INJECTION64_PATH when launching the application.
Quick start
Get the source
Use either the release source archive or a git clone:
Download and extract the source archive from the release page, then build from the extracted directory.
Or clone the repository and initialize its submodules:
git clone https://github.com/NVIDIA/nsight-perf-sdk-for-ai.git cd nsight-perf-sdk-for-ai git submodule update --init --recursive
Build the client, tools, and samples
cmake -S . -B build
cmake --build build -j 8
Build output layout
The build writes executables, libraries, and benchmark runner scripts under build/bin.
build/
└── bin/
├── nvidia-nsight-perf # downloaded service executable
├── libnvperf_grfx_host.so # downloaded nvperf-core library
├── BenchmarkNvTraceCuda
├── BenchmarkNvTraceDemangler
├── BenchmarkNvTraceMemory
├── BenchmarkNvTraceNvtx
├── BenchmarkNvTraceServiceInitialization
├── BenchmarkNvTraceZipStreamingCompressor
├── DataModelValidator
├── TestNvTraceCudaContinuousProfiling
├── UptimeTester
├── nvtrace_continuous_export
├── nvtrace_merge_perfetto_traces
├── nvtrace_perfetto_trace_exporter
├── nvtrace_poll_until_service_ready
├── nvtrace_snapshot_consumer
├── nvtrace_streaming_consumer
├── nvtx_cuda_driver_injection_proxy # shared library
├── BenchmarkCommon.sh
├── RunCudaBenchmarks.sh
├── RunNvtxBenchmarks.sh
├── benchmark and benchmark_main # static libraries
├── flatbuffers # static library
├── gtest # static library
├── libprotobuf, libprotobuf-lite, and libprotoc # static library targets
├── perfetto_trace_proto # static library
└── protoc # native code-generation executable
nvidia-nsight-perf and its nvperf-core library are proprietary binaries, and are downloaded rather than built from source.
Cross-compiled builds create the host-native protoc used during the build under build/third_party/perfetto/protoc_host/bin instead of build/bin.
Choose a workflow
- To verify a new SDK build, run NvTraceContinuousProfiling as a smoke test.
- To profile an existing application without source changes, start the service before the workload and capture with a streaming or snapshot consumer.
- To profile a multi-process or Slurm workload, follow the cluster profiling guidance.
Start the service
From the directory containing the matching service and tool binaries:
./nvidia-nsight-perf &
./nvtrace_poll_until_service_ready
Run the CUDA workload while the service remains active.
Capture and export a trace
Use a streaming capture for a fixed interval:
./nvtrace_streaming_consumer --output-dir . --duration-ms 10000
Or use a lookback snapshot:
./nvtrace_snapshot_consumer --output-dir . --duration-ms 1000
Each consumer writes a Trace Stream Database directory. Export it to a Perfetto trace:
./nvtrace_perfetto_trace_exporter --tsf-database-dir <hostname>.<timestamp>
Open the generated .perfetto-trace.zip file in https://ui.perfetto.dev. If a trace is too large to open directly, compress an uncompressed trace with gzip -9; Perfetto can open .gz files.
For complete command-line details, see the tool documentation. For the sample build and smoke test, see the CUDA samples.
Compatibility
NVIDIA Nsight Perf makes these compatibility guarantees:
- CUDA forward compatibility: NVIDIA Nsight Perf release remains compatible with later CUDA Toolkit, runtime, and driver releases. Features introduced by a later CUDA release may require a newer NVIDIA Nsight Perf release and might not be collected or displayed by an older release.
- Service-consumer compatibility: service-connected NvTrace consumers and the NVIDIA Nsight Perf service are compatible across SDK releases in both directions. A newer consumer can connect to an older service, and an older consumer can connect to a newer service. Cross-version use provides the features understood by the participating versions; it does not make newer features available to an older component.
Software Architecture
NVIDIA Nsight Perf has two primary layers:
- The service runs in a separate process, collects data from GPU and target-side providers, and publishes provider buffers to authorized clients. One service can observe multiple processes and GPUs.
- The client connects to the service, reads provider buffers through observers, writes Trace Streaming Format (
.nvtsf) data blocks, builds an offline data model, and exports that model to formats such as Perfetto.
The profiled workload runs independently from the consumer. Start the service and consumer before the workload enters the region of interest so the observation window contains all relevant activity.
Known issues and limitations
- Each GPU can be initialized by only one NVIDIA Nsight Perf service instance at a time because GPU PM-counter and hardware-event sessions are exclusive.
- The
nvidia-nsight-perfexecutable should load a CUDA user-mode driver that matches the host kernel-mode driver. Otherwise, GPU GR-context tracing may fail, resulting in incomplete GPU-event correlation or missing GPU timings for named CUDA kernels. See GPU GR-context tracing and CUDA forward compatibility for driver requirements and deployment checks. - Kernels that start before the observation window may not appear.
- NVTX start-end ranges are not supported.
- On the 620 driver, mixing the NVTX core push-pop APIs (nvtxRangePush*/nvtxRangePop) and the Payload Extension push-pop APIs (nvtxRangePushPayload/nvtxRangePopPayload) on the same range may cause the push-pop ranges on that thread to be traced incorrectly. Use the NVTX CUDA driver injection proxy to trace such ranges correctly. Driver versions later than 620 do not require this.
- On the 620 driver, using NVTX in a parent before
fork()and in its child afterward can cause incorrect traces, hangs or crashes. Using NVTX on only one side of the fork, or in a child that callsexec()before using NVTX, is supported.- For PyTorch
DataLoaderworkers, usemultiprocessing_context="spawn"ormultiprocessing_context="forkserver"instead of"fork".
- For PyTorch
- The maximum recommended GPU PM-counter sampling rate is 20 kHz.
- For batched asynchronous memcpy (cuMemcpyBatchAsync_v2), NVIDIA Nsight Perf may display multiple copies in a CUDA submission group as one GPU memcpy event. The event's duration covers the group, but its transfer details may describe only the first copy. Do not use these events to determine per-copy counts, sizes, addresses, bandwidth, or the duration of the entire batch.
Contributions
Contributions require signed-off commits and the Developer Certificate of Origin. See CONTRIBUTING.md before submitting a change.
Security
Do not report potential security vulnerabilities through public GitHub issues. Follow the private reporting process in SECURITY.md.
License
NVIDIA-authored aggregate source and build material are licensed under Apache-2.0; see LICENSE. Components, imported CUDA Samples, NVTX, and proprietary binaries retain their own terms. See LICENSING.md and THIRD_PARTY_NOTICES.md for component paths, source repositories, and notices.
External materials
NOTICE AND DISCLAIMER: This software automatically retrieves, accesses or interacts with external materials. Those retrieved materials are not distributed with this software and are governed solely by separate terms, conditions and licenses. You are solely responsible for finding, reviewing and complying with all applicable terms, conditions, and licenses, and for verifying the security, integrity and suitability of any retrieved materials for your specific use case. This software is provided "AS IS", without warranty of any kind. The author makes no representations or warranties regarding any retrieved materials, and assumes no liability for any losses, damages, liabilities or legal consequences from your use or inability to use this software or any retrieved materials. Use this software and the retrieved materials at your own risk.
Component source and license files
| Component path | Source repository | Source license | License file |
|---|---|---|---|
nsight-perf-daemon |
NVIDIA/nsight-perf-daemon | Apache-2.0 | nsight-perf-daemon/LICENSE |
nvperf-core |
NVIDIA/nvperf-core | Apache-2.0 | nvperf-core/LICENSE |
nvtrace-client |
NVIDIA/nvtrace-client | Apache-2.0 | nvtrace-client/LICENSE |
nvtrace-samples-cuda |
NVIDIA/nvtrace-samples-cuda | Apache-2.0 | nvtrace-samples-cuda/LICENSE |
nvtrace-tools |
NVIDIA/nvtrace-tools | Apache-2.0 | nvtrace-tools/LICENSE |
NVTX |
NVIDIA/NVTX | Apache-2.0 WITH LLVM-exception | NVTX/LICENSE.txt |
These independently distributed components retain their own license files. The source licenses above do not cover separately downloaded proprietary binaries or override separately marked assets. See LICENSING.md for those boundaries.
Additional bundled license files:
nvtrace-samples-cuda/cuda-samples/: BSD-style CUDA Samples terms; see nvtrace-samples-cuda/cuda-samples/LICENSE.NVTX/python/: Apache-2.0 WITH LLVM-exception; see NVTX/python/LICENSE.txt.NVTX/rust/: Apache-2.0 WITH LLVM-exception; see NVTX/rust/LICENSE.NVTX/docs/: Apache-2.0 WITH LLVM-exception, subject to separately marked assets; see NVTX/docs/LICENSE.txt.NVTX/tools/docs/github-markdown-css/: MIT; see NVTX/tools/docs/github-markdown-css/license.