[Diagnostics][dotnet-trace] Add collect-linux verb

docs/core/diagnostics/dotnet-trace.md

@@ -43,7 +43,8 @@ The `dotnet-trace` tool:
 * Is a cross-platform .NET Core tool.
 * Enables the collection of .NET Core traces of a running process without a native profiler.
 * Is built on [`EventPipe`](./eventpipe.md) of the .NET Core runtime.
-* Delivers the same experience on Windows, Linux, or macOS.
+* Delivers the same experience on Windows, Linux, and macOS.
+* Offers an additional command on Linux to collect Linux perf events.
 
 ## Options
 
@@ -55,15 +56,12 @@ The `dotnet-trace` tool:
 
   Displays the version of the dotnet-trace utility.
 
-- **`--duration`**
-
-  How long to run the trace. `--duration 00:00:00:05` will run it for 5 seconds.
-
 ## Commands
 
 | Command                                                   |
 |-----------------------------------------------------------|
 | [dotnet-trace collect](#dotnet-trace-collect)             |
+| [dotnet-trace collect-linux](#dotnet-trace-collect-linux) |
 | [dotnet-trace convert](#dotnet-trace-convert)             |
 | [dotnet-trace ps](#dotnet-trace-ps)                       |
 | [dotnet-trace list-profiles](#dotnet-trace-list-profiles) |
@@ -76,13 +74,23 @@ Collects a diagnostic trace from a running process or launches a child process a
 ### Synopsis
 
 ```dotnetcli
-dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--clrevents <clrevents>]
+dotnet-trace collect
+    [--buffersize <size>]
+    [--clreventlevel <clreventlevel>]
+    [--clrevents <clrevents>]
     [--dsrouter <ios|ios-sim|android|android-emu>]
-    [--format <Chromium|NetTrace|Speedscope>] [-h|--help] [--duration dd:hh:mm:ss]
-    [-n, --name <name>] [--diagnostic-port] [-o|--output <trace-file-path>] [-p|--process-id <pid>]
-    [--profile <profile-name>] [--providers <list-of-comma-separated-providers>]
+    [--format <Chromium|NetTrace|Speedscope>]
+    [-h|--help]
+    [--duration dd:hh:mm:ss]
+    [-n, --name <name>]
+    [--diagnostic-port]
+    [-o|--output <trace-file-path>]
+    [-p|--process-id <pid>]
+    [--profiles <list-of-comma-separated-profile-names>]
+    [--providers <list-of-comma-separated-providers>]
     [-- <command>] (for target applications running .NET 5 or later)
-    [--show-child-io] [--resume-runtime]
+    [--show-child-io]
+    [--resume-runtime]
     [--stopping-event-provider-name <stoppingEventProviderName>]
     [--stopping-event-event-name <stoppingEventEventName>]
     [--stopping-event-payload-filter <stoppingEventPayloadFilter>]
@@ -119,7 +127,7 @@ dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--
   | ------------ | ------------------- |
   | `gc` | `0x1` |
   | `gchandle` | `0x2` |
-  | `fusion` | `0x4` |
+  | `assemblyloader` | `0x4` |
   | `loader` | `0x8` |
   | `jit` | `0x10` |
   | `ngen` | `0x20` |
@@ -138,7 +146,7 @@ dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--
   | `gcheapdump` | `0x100000` |
   | `gcsampledobjectallocationhigh` | `0x200000` |
   | `gcheapsurvivalandmovement` | `0x400000` |
-  | `gcheapcollect` | `0x800000` |
+  | `managedheapcollect` | `0x800000` |
   | `gcheapandtypenames` | `0x1000000` |
   | `gcsampledobjectallocationlow` | `0x2000000` |
   | `perftrack` | `0x20000000` |
@@ -152,13 +160,16 @@ dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--
   | `compilationdiagnostic` | `0x2000000000` |
   | `methoddiagnostic` | `0x4000000000` |
   | `typediagnostic` | `0x8000000000` |
+  | `jitinstrumentationdata` | `0x10000000000` |
+  | `profiler` | `0x20000000000` |
   | `waithandle` | `0x40000000000` |
+  | `allocationsampling` | `0x80000000000` |
 
   You can read about the CLR provider more in detail on the [.NET runtime provider reference documentation](../../fundamentals/diagnostics/runtime-events.md).
 
 - **`--dsrouter {ios|ios-sim|android|android-emu}**
 
- Starts [dotnet-dsrouter](dotnet-dsrouter.md) and connects to it. Requires [dotnet-dsrouter](dotnet-dsrouter.md) to be installed. Run `dotnet-dsrouter -h` for more information.
+  Starts [dotnet-dsrouter](dotnet-dsrouter.md) and connects to it. Requires [dotnet-dsrouter](dotnet-dsrouter.md) to be installed. Run `dotnet-dsrouter -h` for more information.
 
 - **`--format {Chromium|NetTrace|Speedscope}`**
 
@@ -200,19 +211,24 @@ dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--
   > [!NOTE]
   > On Linux and macOS, using this option requires the target application and `dotnet-trace` to share the same `TMPDIR` environment variable. Otherwise, the command will time out.
 
-- **`--profile <profile-name>`**
+- **`--profiles <list-of-comma-separated-profile-names>`**
+
+  A comma-separated list of named, pre-defined set of provider configurations for common tracing scenarios. The union of profiles and providers will be in effect.
 
-  A named pre-defined set of provider configurations that allows common tracing scenarios to be specified succinctly. The following profiles are available:
+  Default behavior (when `--profiles`, `--providers`, and `--clrevents` are omitted): dotnet-trace enables a useful, low-overhead composition: `dotnet-common` + `dotnet-thread-time`.
 
- | Profile | Description |
- |---------|-------------|
- |`cpu-sampling`|Useful for tracking CPU usage and general .NET runtime information. This is the default option if no profile or providers are specified.|
- |`gc-verbose`|Tracks GC collections and samples object allocations.|
- |`gc-collect`|Tracks GC collections only at very low overhead.|
+  Available profiles:
+
+  | Profile | Description |
+  |---------|-------------|
+  |`dotnet-common`|Lightweight .NET runtime diagnostics (GC, JIT, loader/assembly loader, threading, contention). Designed to stay low overhead.|
+  |`dotnet-thread-time`|Samples .NET thread stacks (~100 Hz) to identify hotspots over time. Uses the runtime sample profiler with managed stacks.|
+  |`gc-verbose`|Tracks GC collections and samples object allocations.|
+  |`gc-collect`|Tracks GC collections only at very low overhead.|
 
 - **`--providers <list-of-comma-separated-providers>`**
 
-  A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile <profile-name>`. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the profile.
+  A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profiles <list-of-comma-separated-profile-names>`. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the profiles.
 
   This list of providers is in the form:
 
@@ -259,6 +275,205 @@ dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--
 
 > - When specifying a stopping event through the `--stopping-event-*` options, as the EventStream is being parsed asynchronously, there will be some events that pass through between the time a trace event matching the specified stopping event options is parsed and the EventPipeSession is stopped.
 
+## dotnet-trace collect-linux
+
+Collects diagnostic traces using perf_events, a Linux OS technology. `collect-linux` requires admin privileges to capture kernel- and user-mode events, and by default, captures events from all processes.
+
+This Linux-only command includes the same .NET events as [`dotnet-trace collect`](#dotnet-trace-collect), and it uses the kernel’s user_events mechanism to emit .NET events as perf events, enabling unification of user-space .NET events with kernel-space system events.
+
+### Default collection behavior
+
+When `--profiles`, `--clrevents`, and `--providers` aren’t specified, `collect-linux` enables the default `profile` providing the comprehensive composition:
+
+- `dotnet-common` — lightweight .NET runtime diagnostics.
+- `dotnet-thread-time` — runtime thread stack sampling.
+- `kernel-cpu` — kernel CPU sampling (perf-based) via `Universal.Events/cpu`.
+- `kernel-cswitch` — kernel context switches via `Universal.Events/cswitch` for on/off-CPU analysis.
+
+By default, a machine-wide trace will be collected. .NET Processes are discovered through their diagnostics ports, which are located under the `TMPDIR` environment variable when set and otherwise under `/tmp`. `dotnet-trace` will need to share the same `TMPDIR` value as .NET Processes of interest, otherwise events will not be collected from those processes.
+
+If collecting events from all .NET Processes is undesired, `-n, --name <name>` or `-p|--process-id <PID>` can be used to specify a particular process. **NOTE**: Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application.
+
+### Prerequisites
+
+- Linux kernel with `CONFIG_USER_EVENTS=y` support (kernel 6.4+)
+- Root permissions
+- .NET 10+
+
+### Synopsis
+
+```dotnetcli
+dotnet-trace collect-linux
+    [-h|--help]
+
+    # Provider/Event Specification
+    [--providers <list-of-comma-separated-providers>]
+    [--clreventlevel <clreventlevel>]
+    [--clrevents <clrevents>]
+    [--perf-event-tracepoints <list-of-perf-event-tracepoints>]
+    [--profiles <list-of-comma-separated-profile-names>]
+
+    # Trace Collection
+    [--format <Chromium|NetTrace|Speedscope>]
+    [-o|--output <trace-file-path>]
+    [--duration dd:hh:mm:ss]
+
+    # .NET Process Target (Optional)
+    [-n, --name <name>]
+    [-p|--process-id <pid>]
+    [-- <command>]
+    [--show-child-io]
+```
+
+### Options
+
+#### Provider/Event Specification Options
+
+- **`--providers <list-of-comma-separated-providers>`**
+
+  A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profiles <list-of-comma-separated-profile-names>`. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the specified profiles.
+
+  This list of providers is in the form:
+
+  - `Provider[,Provider]`
+  - `Provider` is in the form: `KnownProviderName[:Flags[:Level][:KeyValueArgs]]`.
+  - `KeyValueArgs` is in the form: `[key1=value1][;key2=value2]`.
+
+  To learn more about some of the well-known providers in .NET, refer to [Well-known Event Providers](./well-known-event-providers.md).
+
+- **`--clreventlevel <clreventlevel>`**
+
+  Verbosity of CLR events to be emitted.
+  The following table shows the available event levels.
+
+  | String value    | Numeric value |
+  | --------------- | :-----------: |
+  | `logalways`     |      `0`      |
+  | `critical`      |      `1`      |
+  | `error`         |      `2`      |
+  | `warning`       |      `3`      |
+  | `informational` |      `4`      |
+  | `verbose`       |      `5`      |
+
+- **`--clrevents <clrevents>`**
+
+  A list of CLR runtime provider keywords to enable separated by `+` signs. This is a simple mapping that lets you specify event keywords via string aliases rather than their hex values. For example, `dotnet-trace collect-linux --providers Microsoft-Windows-DotNETRuntime:3:4` requests the same set of events as `dotnet-trace collect-linux --clrevents gc+gchandle --clreventlevel informational`. The table below shows the list of available keywords:
+
+  | Keyword String Alias | Keyword Hex Value |
+  | ------------ | ------------------- |
+  | `gc` | `0x1` |
+  | `gchandle` | `0x2` |
+  | `assemblyloader` | `0x4` |
+  | `loader` | `0x8` |
+  | `jit` | `0x10` |
+  | `ngen` | `0x20` |
+  | `startenumeration` | `0x40` |
+  | `endenumeration` | `0x80` |
+  | `security` | `0x400` |
+  | `appdomainresourcemanagement` | `0x800` |
+  | `jittracing` | `0x1000` |
+  | `interop` | `0x2000` |
+  | `contention` | `0x4000` |
+  | `exception` | `0x8000` |
+  | `threading` | `0x10000` |
+  | `jittedmethodiltonativemap` | `0x20000` |
+  | `overrideandsuppressngenevents` | `0x40000` |
+  | `type` | `0x80000` |
+  | `gcheapdump` | `0x100000` |
+  | `gcsampledobjectallocationhigh` | `0x200000` |
+  | `gcheapsurvivalandmovement` | `0x400000` |
+  | `managedheapcollect` | `0x800000` |
+  | `gcheapandtypenames` | `0x1000000` |
+  | `gcsampledobjectallocationlow` | `0x2000000` |
+  | `perftrack` | `0x20000000` |
+  | `stack` | `0x40000000` |
+  | `threadtransfer` | `0x80000000` |
+  | `debugger` | `0x100000000` |
+  | `monitoring` | `0x200000000` |
+  | `codesymbols` | `0x400000000` |
+  | `eventsource` | `0x800000000` |
+  | `compilation` | `0x1000000000` |
+  | `compilationdiagnostic` | `0x2000000000` |
+  | `methoddiagnostic` | `0x4000000000` |
+  | `typediagnostic` | `0x8000000000` |
+  | `jitinstrumentationdata` | `0x10000000000` |
+  | `profiler` | `0x20000000000` |
+  | `waithandle` | `0x40000000000` |
+  | `allocationsampling` | `0x80000000000` |
+
+  You can read about the CLR provider more in detail on the [.NET runtime provider reference documentation](../../fundamentals/diagnostics/runtime-events.md).
+
+- **`--perf-event-tracepoints <list-of-perf-event-tracepoints>`**
+
+  A comma-separated list of kernel perf event tracepoints to include in the trace. Available kernel events can be found under tracefs, which is typically mounted at `/sys/kernel/tracing`, through `available_events` for all available tracepoints or through the `events/` subdirectory for categorized events.
+
+  Example: `--perf-event-tracepoints syscalls:sys_enter_execve,sched:sched_switch,sched:sched_wakeup`
+
+- **`--profiles <list-of-comma-separated-profile-names>`**
+
+  A named, pre-defined set of provider configurations for common tracing scenarios. You can specify multiple profiles as a comma-separated list. When multiple profiles are specified, the providers and settings are combined (union), and duplicates are ignored.
+
+  Available profiles:
+
+  | Profile | Description |
+  |---------|-------------|
+  |`dotnet-common`|Lightweight .NET runtime diagnostics (GC, JIT, loader/assembly loader, threading, contention). Designed to stay low overhead.|
+  |`dotnet-thread-time`|Samples .NET thread stacks (~100 Hz) to identify hotspots over time. Uses the runtime sample profiler with managed stacks.|
+  |`kernel-cpu`|Kernel CPU sampling (perf-based), emitted as `Universal.Events/cpu`, for precise on-CPU attribution.|
+  |`kernel-cswitch`|Kernel thread context switches, emitted as `Universal.Events/cswitch`, for on/off-CPU and scheduler analysis.|
+  |`kernel-precise-thread-time`|Combines `kernel-cpu` + `kernel-cswitch` for a precise thread-time view using kernel events only.|
+  |`gc-verbose`|Tracks GC collections and samples object allocations.|
+  |`gc-collect`|Tracks GC collections only at very low overhead.|
+
+  > [!NOTE]
+  > The former default `cpu-sampling` profile is now named as `dotnet-thread-time`.
+
+#### Trace Collection Options
+
+- **`--format {Chromium|NetTrace|Speedscope}`**
+
+  Sets the output format for the trace file conversion. The default is `NetTrace`.
+
+- **`-o|--output <trace-file-path>`**
+
+  The output path for the collected trace data. If not specified it defaults to `<appname>_<yyyyMMdd>_<HHmmss>.nettrace`, for example, `myapp_20210315_111514.nettrace``.
+
+- **`--duration <time-to-run>`**
+
+  The time for the trace to run. Use the `dd:hh:mm:ss` format. For example `00:00:00:05` will run it for 5 seconds.
+
+#### .NET Process Target Options
+
+See [Default collection behavior](#default-collection-behavior)
+
+- **`-n, --name <name>`**
+
+  The name of the process to collect the trace from.
+
+  > [!NOTE]
+  > Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application.
+
+- **`-p|--process-id <PID>`**
+
+  The process ID to collect the trace from.
+
+  > [!NOTE]
+  > Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application.
+
+- **`-- <command>`**
+
+  After the collection configuration parameters, the user can append `--` followed by a command to start a .NET application. This may be helpful when diagnosing issues that happen early in the process, such as startup performance issue or assembly loader and binder errors.
+
+- **`--show-child-io`**
+
+  Only applicable when `-- <command>` is used. Shows the input and output streams of a launched child process in the current console.
+
+> [!NOTE]
+
+> - Stopping the trace may take a long time (up to minutes) for large applications. The runtime needs to send over the type cache for all managed code that was captured in the trace.
+
+> - To collect a trace using `dotnet-trace collect-linux`, it needs to be run with root permissions (`CAP_PERFMON`/`CAP_SYS_ADMIN`). Otherwise, the tool will fail to collect events.
+
 ## dotnet-trace convert
 
 Converts `nettrace` traces to alternate formats for use with alternate trace analysis tools.
@@ -406,11 +621,11 @@ dotnet-trace collect -- hello.exe arg1 arg2
 The preceding command generates output similar to the following:
 
 ```output
-No profile or providers specified, defaulting to trace profile 'cpu-sampling'
+No profile or providers specified. Using default composition: dotnet-common + dotnet-thread-time
 
 Provider Name                           Keywords            Level               Enabled By
-Microsoft-DotNETCore-SampleProfiler     0x0000F00000000000  Informational(4)    --profile
-Microsoft-Windows-DotNETRuntime         0x00000014C14FCCBD  Informational(4)    --profile
+Microsoft-DotNETCore-SampleProfiler     0x0000F00000000000  Informational(4)    --profiles
+Microsoft-Windows-DotNETRuntime         0x00000014C14FCCBD  Informational(4)    --profiles
 
 Process        : E:\temp\gcperfsim\bin\Debug\net5.0\gcperfsim.exe
 Output File    : E:\temp\gcperfsim\trace.nettrace