diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index b2f6512906ca9..67893c219a8c3 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -43,7 +43,18 @@ The `dotnet-trace` tool: * Is a cross-platform .NET Core tool. * Enables the collection of .NET Core traces of a running process without a native profiler. * Is built on [`EventPipe`](./eventpipe.md) of the .NET Core runtime. -* Delivers the same experience on Windows, Linux, or macOS. +* Supports two different ways of collecting traces: + + - The [`collect` verb](#dotnet-trace-collect) offers consistent functionality on any OS. + - The [`collect-linux` verb](#dotnet-trace-collect-linux) uses Linux-specific OS capabilities to provide additional features. + + | Characteristic | `collect` | `collect-linux` | + |------------------------------------------|----------|-----------------------------------| + | Supported OS | Any | Linux only, kernel version >= 6.4 | + | Requires Admin/Root Privilege | No | Yes | + | Trace all processes simultaneously | No | Supported | + | Capture native library and kernel events | No | Supported | + | Event callstacks include native frames | No | Yes | ## Options @@ -55,15 +66,12 @@ The `dotnet-trace` tool: Displays the version of the dotnet-trace utility. -- **`--duration`** - - How long to run the trace. `--duration 00:00:00:05` will run it for 5 seconds. - ## Commands | Command | |-----------------------------------------------------------| | [dotnet-trace collect](#dotnet-trace-collect) | +| [dotnet-trace collect-linux](#dotnet-trace-collect-linux) | | [dotnet-trace convert](#dotnet-trace-convert) | | [dotnet-trace ps](#dotnet-trace-ps) | | [dotnet-trace list-profiles](#dotnet-trace-list-profiles) | @@ -76,13 +84,23 @@ Collects a diagnostic trace from a running process or launches a child process a ### Synopsis ```dotnetcli -dotnet-trace collect [--buffersize ] [--clreventlevel ] [--clrevents ] +dotnet-trace collect + [--buffersize ] + [--clreventlevel ] + [--clrevents ] [--dsrouter ] - [--format ] [-h|--help] [--duration dd:hh:mm:ss] - [-n, --name ] [--diagnostic-port] [-o|--output ] [-p|--process-id ] - [--profile ] [--providers ] + [--format ] + [-h|--help] + [--duration dd:hh:mm:ss] + [-n, --name ] + [--diagnostic-port] + [-o|--output ] + [-p|--process-id ] + [--profile ] + [--providers ] [-- ] (for target applications running .NET 5 or later) - [--show-child-io] [--resume-runtime] + [--show-child-io] + [--resume-runtime] [--stopping-event-provider-name ] [--stopping-event-event-name ] [--stopping-event-payload-filter ] @@ -99,7 +117,7 @@ dotnet-trace collect [--buffersize ] [--clreventlevel ] [-- - **`--clreventlevel `** - Verbosity of CLR events to be emitted. + Verbosity of CLR events to be emitted. This option only applies when `--clrevents` is specified and not overridden by `--profile` or `--providers`. The following table shows the available event levels. | String value | Numeric value | @@ -113,13 +131,13 @@ dotnet-trace collect [--buffersize ] [--clreventlevel ] [-- - **`--clrevents `** - A list of CLR runtime provider keywords to enable separated by `+` signs. This is a simple mapping that lets you specify event keywords via string aliases rather than their hex values. For example, `dotnet-trace collect --providers Microsoft-Windows-DotNETRuntime:3:4` requests the same set of events as `dotnet-trace collect --clrevents gc+gchandle --clreventlevel informational`. The table below shows the list of available keywords: + A list of CLR runtime provider keywords to enable separated by `+` signs. This is a simple mapping that lets you specify event keywords via string aliases rather than their hex values. For example, `dotnet-trace collect --providers Microsoft-Windows-DotNETRuntime:3:4` requests the same set of events as `dotnet-trace collect --clrevents gc+gchandle --clreventlevel informational`. If the CLR runtime provider `Microsoft-Windows-DotNETRuntime` is also enabled through `--providers` or `--profile`, this option is ignored. The following table shows the list of available keywords: | Keyword String Alias | Keyword Hex Value | | ------------ | ------------------- | | `gc` | `0x1` | | `gchandle` | `0x2` | - | `fusion` | `0x4` | + | `assemblyloader` | `0x4` | | `loader` | `0x8` | | `jit` | `0x10` | | `ngen` | `0x20` | @@ -138,7 +156,7 @@ dotnet-trace collect [--buffersize ] [--clreventlevel ] [-- | `gcheapdump` | `0x100000` | | `gcsampledobjectallocationhigh` | `0x200000` | | `gcheapsurvivalandmovement` | `0x400000` | - | `gcheapcollect` | `0x800000` | + | `managedheapcollect` | `0x800000` | | `gcheapandtypenames` | `0x1000000` | | `gcsampledobjectallocationlow` | `0x2000000` | | `perftrack` | `0x20000000` | @@ -152,13 +170,16 @@ dotnet-trace collect [--buffersize ] [--clreventlevel ] [-- | `compilationdiagnostic` | `0x2000000000` | | `methoddiagnostic` | `0x4000000000` | | `typediagnostic` | `0x8000000000` | + | `jitinstrumentationdata` | `0x10000000000` | + | `profiler` | `0x20000000000` | | `waithandle` | `0x40000000000` | + | `allocationsampling` | `0x80000000000` | - You can read about the CLR provider more in detail on the [.NET runtime provider reference documentation](../../fundamentals/diagnostics/runtime-events.md). + You can read about the CLR provider in more detail on the [.NET runtime provider reference documentation](../../fundamentals/diagnostics/runtime-events.md). - **`--dsrouter {ios|ios-sim|android|android-emu}** - Starts [dotnet-dsrouter](dotnet-dsrouter.md) and connects to it. Requires [dotnet-dsrouter](dotnet-dsrouter.md) to be installed. Run `dotnet-dsrouter -h` for more information. + Starts [dotnet-dsrouter](dotnet-dsrouter.md) and connects to it. Requires [dotnet-dsrouter](dotnet-dsrouter.md) to be installed. Run `dotnet-dsrouter -h` for more information. - **`--format {Chromium|NetTrace|Speedscope}`** @@ -200,25 +221,33 @@ dotnet-trace collect [--buffersize ] [--clreventlevel ] [-- > [!NOTE] > On Linux and macOS, using this option requires the target application and `dotnet-trace` to share the same `TMPDIR` environment variable. Otherwise, the command will time out. -- **`--profile `** +- **`--profile `** + + A profile is a predefined set of provider configurations for common tracing scenarios. Multiple profiles can be specified at a time, delimited by commas. Providers configured through `--providers` override the profile's configuration. Similarly, if any profile configures the CLR runtime provider, it will override any configurations prescribed through `--clrevents`. - A named pre-defined set of provider configurations that allows common tracing scenarios to be specified succinctly. The following profiles are available: + When `--profile`, `--providers`, and `--clrevents` are all omitted, `dotnet-trace collect` enables profiles `dotnet-common` and `dotnet-sampled-thread-time` by default. - | Profile | Description | - |---------|-------------| - |`cpu-sampling`|Useful for tracking CPU usage and general .NET runtime information. This is the default option if no profile or providers are specified.| - |`gc-verbose`|Tracks GC collections and samples object allocations.| - |`gc-collect`|Tracks GC collections only at very low overhead.| + Available profiles: + + | Profile | Description | + |---------|-------------| + |`dotnet-common`|Lightweight .NET runtime diagnostics designed to stay low overhead.
Includes GC, AssemblyLoader, Loader, JIT, Exceptions, Threading, JittedMethodILToNativeMap, and Compilation events
Equivalent to `--providers "Microsoft-Windows-DotNETRuntime:0x100003801D:4"`.| + |`dotnet-sampled-thread-time`|Samples .NET thread stacks (~100 Hz) to identify hotspots over time. Uses the runtime sample profiler with managed stacks.| + |`gc-verbose`|Tracks GC collections and samples object allocations.| + |`gc-collect`|Tracks GC collections only at very low overhead.| + |`database`|Captures ADO.NET and Entity Framework database commands.| + + > [!NOTE] + > In past versions of the dotnet-trace tool, the collect verb supported a profile called `cpu-sampling`. This profile was removed because the name was misleading. It sampled all threads regardless of their CPU usage. You can achieve a similar result now using `--profile dotnet-sampled-thread-time,dotnet-common`. If you need to match the former `cpu-sampling` behavior exactly, use `--profile dotnet-sampled-thread-time --providers "Microsoft-Windows-DotNETRuntime:0x14C14FCCBD:4"`. - **`--providers `** - A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the profile. + A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from `--profile` and `--clrevents`. - This list of providers is in the form: + This list of providers is in the form `Provider[,Provider]`: - - `Provider[,Provider]` - - `Provider` is in the form: `KnownProviderName[:Flags[:Level][:KeyValueArgs]]`. - - `KeyValueArgs` is in the form: `[key1=value1][;key2=value2]`. + - `Provider` is in the form: `KnownProviderName[:Flags[:Level[:KeyValueArgs]]]` + - `KeyValueArgs` is in the form: `[key1=value1][;key2=value2]` To learn more about some of the well-known providers in .NET, refer to [Well-known Event Providers](./well-known-event-providers.md). @@ -259,6 +288,200 @@ dotnet-trace collect [--buffersize ] [--clreventlevel ] [-- > - When specifying a stopping event through the `--stopping-event-*` options, as the EventStream is being parsed asynchronously, there will be some events that pass through between the time a trace event matching the specified stopping event options is parsed and the EventPipeSession is stopped. +## dotnet-trace collect-linux + +> [!NOTE] +> The `collect-linux` verb is a new preview feature and relies on an updated version of the .nettrace file format. The latest PerfView release supports these trace files, but other ways of using the trace file, such as [`convert`](#dotnet-trace-convert) and [`report`](#dotnet-trace-report), might not work yet. + +Collects diagnostic traces using perf_events, a Linux OS technology. `collect-linux` enables the following additional features over [`collect`](#dotnet-trace-collect). + +| | collect | collect-linux | +|------------------------------------------|----------|-----------------------------------| +| Supported OS | Any | Linux only, kernel version >= 6.4 | +| Requires Admin/Root privilege | No | Yes | +| Trace all processes simultaneously | No | Supported | +| Capture native library and kernel events | No | Supported | +| Event callstacks include native frames | No | Yes | + +### Prerequisites + +- Linux kernel with `CONFIG_USER_EVENTS=y` support (kernel 6.4+) +- Root permissions +- .NET 10+ + +> [!NOTE] +> The `collect-linux` verb only runs on linux x64 and linux arm64 environments that have glibc version 2.35 or above. +> All of the [.NET 10 officially supported Linux distros](https://github.com/dotnet/core/blob/main/release-notes/10.0/supported-os.md#linux) support this requirement except Alpine 3.22, CentOS Stream 9, and any distros based off Red Hat Enterprise Linux 9. +> A quick way to check the version of a system's libc is with the command `ldd --version` or by executing the libc library directly. + +### Synopsis + +```dotnetcli +dotnet-trace collect-linux + [-h|--help] + + # Provider/Event Specification + [--providers ] + [--clreventlevel ] + [--clrevents ] + [--perf-events ] + [--profile ] + + # Trace Collection + [-o|--output ] + [--duration dd:hh:mm:ss] + + # .NET Process Target (Optional) + [-n, --name ] + [-p|--process-id ] + + # Probe mode + [--probe] +``` + +### Default collection behavior + +When `--providers`, `--profile`, `--clrevents`, and `--perf-events` aren’t specified, `collect-linux` enables these profiles by default: + +- `dotnet-common` — lightweight .NET runtime diagnostics. +- `cpu-sampling` — kernel CPU sampling. + +By default, all processes on the machine are traced. To trace only one process, use `-n, --name ` or `-p|--process-id `. + +### Options + +#### Provider/Event Specification Options + +- **`--providers `** + + A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from `--profile` and `--clrevents`. + + This list of providers is in the form `Provider[,Provider]`: + + - `Provider` is in the form: `KnownProviderName[:Flags[:Level[:KeyValueArgs]]]` + - `KeyValueArgs` is in the form: `[key1=value1][;key2=value2]` + + To learn more about some of the well-known providers in .NET, see [Well-known Event Providers](./well-known-event-providers.md). + +- **`--clreventlevel `** + + Verbosity of CLR events to be emitted. This option only applies when `--clrevents` is specified and not overridden by `--profile` or `--providers`. + The following table shows the available event levels. + + | String value | Numeric value | + | --------------- | :-----------: | + | `logalways` | `0` | + | `critical` | `1` | + | `error` | `2` | + | `warning` | `3` | + | `informational` | `4` | + | `verbose` | `5` | + +- **`--clrevents `** + + A list of CLR runtime provider keywords to enable separated by `+` signs. This is a simple mapping that lets you specify event keywords via string aliases rather than their hex values. For example, `dotnet-trace collect-linux --providers Microsoft-Windows-DotNETRuntime:3:4` requests the same set of events as `dotnet-trace collect-linux --clrevents gc+gchandle --clreventlevel informational`. If the CLR runtime provider `Microsoft-Windows-DotNETRuntime` is also enabled through `--providers` or `--profile`, this option is ignored. The following table shows the list of available keywords: + + | Keyword String Alias | Keyword Hex Value | + | ------------ | ------------------- | + | `gc` | `0x1` | + | `gchandle` | `0x2` | + | `assemblyloader` | `0x4` | + | `loader` | `0x8` | + | `jit` | `0x10` | + | `ngen` | `0x20` | + | `startenumeration` | `0x40` | + | `endenumeration` | `0x80` | + | `security` | `0x400` | + | `appdomainresourcemanagement` | `0x800` | + | `jittracing` | `0x1000` | + | `interop` | `0x2000` | + | `contention` | `0x4000` | + | `exception` | `0x8000` | + | `threading` | `0x10000` | + | `jittedmethodiltonativemap` | `0x20000` | + | `overrideandsuppressngenevents` | `0x40000` | + | `type` | `0x80000` | + | `gcheapdump` | `0x100000` | + | `gcsampledobjectallocationhigh` | `0x200000` | + | `gcheapsurvivalandmovement` | `0x400000` | + | `managedheapcollect` | `0x800000` | + | `gcheapandtypenames` | `0x1000000` | + | `gcsampledobjectallocationlow` | `0x2000000` | + | `perftrack` | `0x20000000` | + | `stack` | `0x40000000` | + | `threadtransfer` | `0x80000000` | + | `debugger` | `0x100000000` | + | `monitoring` | `0x200000000` | + | `codesymbols` | `0x400000000` | + | `eventsource` | `0x800000000` | + | `compilation` | `0x1000000000` | + | `compilationdiagnostic` | `0x2000000000` | + | `methoddiagnostic` | `0x4000000000` | + | `typediagnostic` | `0x8000000000` | + | `jitinstrumentationdata` | `0x10000000000` | + | `profiler` | `0x20000000000` | + | `waithandle` | `0x40000000000` | + | `allocationsampling` | `0x80000000000` | + + You can read about the CLR provider in more detail on the [.NET runtime provider reference documentation](../../fundamentals/diagnostics/runtime-events.md). + +- **`--perf-events `** + + A comma-separated list of perf events to include in the trace. Available events can be found under [tracefs](https://www.kernel.org/doc/html/latest/trace/ftrace.html#the-file-system), which is typically mounted at `/sys/kernel/tracing`, through `available_events` for all available events or through the `events/` subdirectory for categorized events. + + Example: `--perf-events syscalls:sys_enter_execve,sched:sched_switch,sched:sched_wakeup` + +- **`--profile `** + + A profile is a predefined set of provider configurations for common tracing scenarios. Multiple profiles can be specified at a time, delimited by commas. Providers configured through `--providers` override the profile's configuration. Similarly, if any profile configures the CLR runtime provider, it will override any configurations prescribed through `--clrevents`. + + When `--profile`, `--providers`, `--clrevents`, and `--perf-events` are all omitted, `dotnet-trace collect-linux` enables profiles `dotnet-common` and `cpu-sampling` by default. + + Available profiles: + + | Profile | Description | + |---------|-------------| + |`dotnet-common`|Lightweight .NET runtime diagnostics designed to stay low overhead.
Includes GC, AssemblyLoader, Loader, JIT, Exceptions, Threading, JittedMethodILToNativeMap, and Compilation events
Equivalent to `--providers "Microsoft-Windows-DotNETRuntime:0x100003801D:4"`.| + |`cpu-sampling`|Kernel CPU sampling (perf-based), emitted as `Universal.Events/cpu`, for precise on-CPU attribution.| + |`thread-time`|Kernel thread context switches, emitted as `Universal.Events/cswitch`, for on/off-CPU and scheduler analysis.| + |`gc-verbose`|Tracks GC collections and samples object allocations.| + |`gc-collect`|Tracks GC collections only at very low overhead.| + |`database`|Captures ADO.NET and Entity Framework database commands.| + +#### Trace Collection Options + +- **`-o|--output `** + + The output path for the collected trace data. If not specified, it defaults to `trace__.nettrace` for the default machine-wide trace and to `__.nettrace` for a process-specific trace (`--name` or `--process-id`) + +- **`--duration `** + + The time for the trace to run. Use the `dd:hh:mm:ss` format. For example `00:00:00:05` will run it for 5 seconds. + +#### .NET Process Target Options + +See [Default collection behavior](#default-collection-behavior) + +- **`-n, --name `** + + The name of the process to collect the trace from. + +- **`-p|--process-id `** + + The process ID to collect the trace from. + +#### Probe mode options + +- **`--probe [-n|--name] [-p|--process-id] [-o|--output ]`** + + Probe .NET processes for support of the EventPipe UserEvents IPC command used by collect-linux, without collecting a trace. Results list supported processes first. Use '-o stdout' to print CSV (pid,processName,supportsCollectLinux) to the console, or '-o output-filename' to write the CSV. Probe a single process with -n|--name or -p|--process-id. + + As running `collect-linux` in probe mode does not collect a trace, it does not require root permissions to run. It does not provide validation of the [prerequisites](#prerequisites), and .NET processes running on preview versions of .NET Runtime '10.0.0' are considered unsupported. + +> [!NOTE] + +> To collect a trace using `dotnet-trace collect-linux`, it needs to be run with root permissions (`CAP_PERFMON`/`CAP_SYS_ADMIN`). Otherwise, the tool will fail to collect events. + ## dotnet-trace convert Converts `nettrace` traces to alternate formats for use with alternate trace analysis tools. @@ -367,7 +590,7 @@ Output the parameters of each method in full. If not specified, parameters will ## Collect a trace with dotnet-trace -To collect traces using `dotnet-trace`: +To collect traces using `dotnet-trace collect`: - Get the process identifier (PID) of the .NET Core application to collect traces from. @@ -384,14 +607,22 @@ To collect traces using `dotnet-trace`: The preceding command generates output similar to the following: ```output - Press to exit... - Connecting to process: /dotnet.exe - Collecting to file: /trace.nettrace - Session Id: - Recording trace 721.025 (KB) + No profile or providers specified, defaulting to trace profiles 'dotnet-common' + 'dotnet-sampled-thread-time'. + + Provider Name Keywords Level Enabled By + Microsoft-Windows-DotNETRuntime 0x000000100003801D Informational(4) --profile + Microsoft-DotNETCore-SampleProfiler 0x0000F00000000000 Informational(4) --profile + + Process : + Output File : _20251007_154557.nettrace + [00:00:00:02] Recording trace 178.172 (KB) + Press or to exit... + Stopping the trace. This may take several minutes depending on the application being traced. + + Trace completed. ``` -- Stop collection by pressing the `` key. `dotnet-trace` will finish logging events to the *trace.nettrace* file. +- Stop collection by pressing the Enter key. `dotnet-trace` will finish logging events to the `.nettrace` file. ## Launch a child application and collect a trace from its startup using dotnet-trace @@ -406,11 +637,11 @@ dotnet-trace collect -- hello.exe arg1 arg2 The preceding command generates output similar to the following: ```output -No profile or providers specified, defaulting to trace profile 'cpu-sampling' +No profile or providers specified, defaulting to trace profiles 'dotnet-common' + 'dotnet-sampled-thread-time'. Provider Name Keywords Level Enabled By +Microsoft-Windows-DotNETRuntime 0x000000100003801D Informational(4) --profile Microsoft-DotNETCore-SampleProfiler 0x0000F00000000000 Informational(4) --profile -Microsoft-Windows-DotNETRuntime 0x00000014C14FCCBD Informational(4) --profile Process : E:\temp\gcperfsim\bin\Debug\net5.0\gcperfsim.exe Output File : E:\temp\gcperfsim\trace.nettrace @@ -466,6 +697,55 @@ However, when you want to gain a finer control over the lifetime of the app bein > [!IMPORTANT] > Launching your app with `dotnet run` can be problematic because the dotnet CLI may spawn many child processes that are not your app and they can connect to `dotnet-trace` before your app, leaving your app to be suspended at run time. It is recommended you directly use a self-contained version of the app or use `dotnet exec` to launch the application. +## (Linux-only) Collect a machine-wide trace using dotnet-trace + +This example captures CPU samples for all processes on the machine. Any processes running .NET 10+ will also include some additional lightweight events describing GC, JIT, and Assembly loading behavior. + + ```output + $ sudo dotnet-trace collect-linux + ========================================================================================== + The collect-linux verb is a new preview feature and relies on an updated version of the + .nettrace file format. The latest PerfView release supports these trace files but other + ways of using the trace file may not work yet. For more details, see the docs at + https://learn.microsoft.com/dotnet/core/diagnostics/dotnet-trace. + ========================================================================================== + No providers, profiles, ClrEvents, or PerfEvents were specified, defaulting to trace profiles 'dotnet-common' + 'cpu-sampling'. + + Provider Name Keywords Level Enabled By + Microsoft-Windows-DotNETRuntime 0x000000100003801D Informational(4) --profile + + Linux Perf Events Enabled By + cpu-sampling --profile + + Output File : trace_20251008_181939.nettrace + + [00:00:00:03] Recording trace. + Press or to exit... + + Recording stopped. + Resolving symbols. + Finished recording trace. + Trace written to trace_20251008_181939.nettrace + ``` + +For environments with multiple .NET versions installed, running `collect-linux` in [probe mode](#probe-mode-options) helps discern whether a .NET process is capable of being traced with collect-linux. + + ```output + $ dotnet-trace collect-linux --probe + ========================================================================================== + The collect-linux verb is a new preview feature and relies on an updated version of the + .nettrace file format. The latest PerfView release supports these trace files but other + ways of using the trace file may not work yet. For more details, see the docs at + https://learn.microsoft.com/dotnet/core/diagnostics/dotnet-trace. + ========================================================================================== + Probing .NET processes for support of the EventPipe UserEvents IPC command used by collect-linux. Requires runtime '10.0.0' or later. + .NET processes that support the command: + 3802935 MyApp + + .NET processes that do NOT support the command: + 3809123 dotnet - Detected runtime: '10.0.0-rc.1.25451.107' + ``` + ## View the trace captured from dotnet-trace On Windows, you can view *.nettrace* files in [Visual Studio](/visualstudio/profiling/beginners-guide-to-performance-profiling?#step-2-analyze-cpu-usage-data) or [PerfView](https://github.com/microsoft/perfview) for analysis.