You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+84-42Lines changed: 84 additions & 42 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,19 +9,25 @@ Intel® PerfSpect is a command-line tool designed to help you analyze and opt
9
9
We welcome bug reports and enhancement requests, which can be submitted via the [Issues](https://github.com/intel/PerfSpect/issues) section on GitHub. For those interested in contributing to the code, please refer to the guidelines outlined in the [CONTRIBUTING.md](CONTRIBUTING.md) file.
10
10
11
11
## Getting PerfSpect
12
+
12
13
Pre-built PerfSpect releases are available in the repository's [Releases](https://github.com/intel/PerfSpect/releases). Download and extract perfspect.tgz.
13
-
```
14
+
15
+
```text
14
16
wget -qO- https://github.com/intel/PerfSpect/releases/latest/download/perfspect.tgz | tar -xvz
15
17
cd perfspect
16
18
```
19
+
17
20
## Running PerfSpect
21
+
18
22
PerfSpect includes a suite of commands designed to analyze and optimize both system and software performance.
19
-
<pre>
23
+
24
+
```text
20
25
Usage:
21
26
perfspect [command] [flags]
22
-
</pre>
27
+
```
23
28
24
29
### Commands
30
+
25
31
| Command | Description |
26
32
| ------- | ----------- |
27
33
|[`metrics`](#metrics-command)| CPU core and uncore metrics |
@@ -36,71 +42,82 @@ Usage:
36
42
> Run `perfspect [command] -h` to view command-specific help text.
37
43
38
44
#### Metrics Command
45
+
39
46
The `metrics` command generates reports containing CPU architectural performance characterization metrics in HTML and CSV formats. Run `perfspect metrics`.
40
47
41
48

42
49
43
50
##### Live Metrics
51
+
44
52
The `metrics` command supports two modes -- default and "live". Default mode behaves as above -- metrics are collected and saved into report files for review. The "live" mode prints the metrics to stdout where they can be viewed in the console and/or redirected into a file or observability pipeline. Run `perfspect metrics --live`.
45
53
46
54

47
55
48
56
##### Metrics Without Root Permissions
57
+
49
58
If neither sudo nor root access is available, an administrator must apply the following configuration to the target system(s):
59
+
50
60
- sysctl -w kernel.perf_event_paranoid=0
51
61
- sysctl -w kernel.nmi_watchdog=0
52
62
- write '125' to all perf_event_mux_interval_ms files found under /sys/devices/*, for example, `for i in $(find /sys/devices -name perf_event_mux_interval_ms); do echo 125 > $i; done`
53
63
54
64
Once the configuration changes are applied, use the `--noroot` flag on the command line, for example, `perfspect metrics --noroot`.
55
65
56
66
##### Refining Metrics to a Specific Time Range
67
+
57
68
After collecting metrics, you can generate new summary reports for a specific time interval using the `metrics trim` subcommand. This is useful when you've collected metrics for an entire workload but want to analyze only a specific portion, excluding setup, teardown, or other unwanted phases.
58
69
59
70
The time range can be specified using either absolute timestamps (seconds since epoch) or relative offsets from the beginning/end of the data. At least one time parameter must be specified.
60
71
61
72
The trimmed CSV and HTML summary files will be placed in a new output directory. The output directory can be specified using the `--output` flag.
62
73
63
74
**Examples:**
64
-
<pre>
75
+
76
+
```text
65
77
# Skip the first 10 seconds and last 5 seconds
66
78
$ ./perfspect metrics trim --input perfspect_2025-11-28_09-21-56 --start-offset 10 --end-offset 5
67
79
68
80
# Use absolute timestamps (seconds since epoch)
69
81
$ ./perfspect metrics trim --input perfspect_2025-11-28_09-21-56 --start-time 1764174327 --end-time 1764174351
70
-
</pre>
82
+
```
71
83
72
84
##### Prometheus Endpoint
85
+
73
86
The `metrics` command can expose metrics via a Prometheus compatible `metrics` endpoint. This allows integration with Prometheus monitoring systems. To enable the Prometheus endpoint, use the `--prometheus-server` flag. By default, the endpoint listens on port 9090. The port can be changed using the `--prometheus-server-addr` flag. Run `perfspect metrics --prometheus-server`. See the [example daemonset](docs/perfspect-daemonset.md) for deploying in Kubernetes.
74
87
75
88
See `perfspect metrics -h` for the extensive set of options and examples.
76
89
77
90
#### Report Command
91
+
78
92
The `report` command generates system configuration reports in a variety of formats. All categories of information are collected by default. See `perfspect report -h` for all options.
79
93
80
94

81
95
82
96
It's possible to report a subset of information by providing command line options. Note that by specifying only the `txt` format, it is printed to stdout, as well as written to a report file.
83
-
<pre>
97
+
98
+
```text
84
99
$ ./perfspect report --bios --format txt
85
100
BIOS
86
101
====
87
102
Vendor: Intel Corporation
88
103
Version: EGSDCRB1.SYS.1752.P05.2401050248
89
104
Release Date: 01/05/2024
90
-
</pre>
105
+
```
91
106
92
107
#### Benchmark Command
108
+
93
109
The `benchmark` command runs performance micro-benchmarks to evaluate system health and performance characteristics. All benchmarks are run by default unless specific benchmarks are selected. A brief system summary is included in the output by default.
94
110
95
111
> [!IMPORTANT]
96
112
> Benchmarks should be run on idle systems to ensure accurate measurements and to avoid interfering with active workloads.
97
113
98
114
**Examples:**
99
-
<pre>
100
-
$ ./perfspect benchmark # Run all benchmarks with system summary
101
-
$ ./perfspect benchmark --speed --power # Run specific benchmarks
102
-
$ ./perfspect benchmark --no-summary # Exclude system summary from output
103
-
</pre>
115
+
116
+
```text
117
+
./perfspect benchmark # Run all benchmarks with system summary
118
+
./perfspect benchmark --speed --power # Run specific benchmarks
119
+
./perfspect benchmark --no-summary # Exclude system summary from output
120
+
```
104
121
105
122
See `perfspect benchmark -h` for all options.
106
123
@@ -109,50 +126,62 @@ See `perfspect benchmark -h` for all options.
109
126
| speed | runs each [stress-ng](https://github.com/ColinIanKing/stress-ng) cpu-method for 1s each, reports the geo-metric mean of all results. |
110
127
| power | runs stress-ng to load all cpus to 100% for 60s. Uses [turbostat](https://github.com/torvalds/linux/tree/master/tools/power/x86/turbostat) to measure power. |
111
128
| temperature | runs the same micro benchmark as 'power', but extracts maximum temperature from turbostat output. |
112
-
| frequency | runs [avx-turbo](https://github.com/travisdowns/avx-turbo) to measure scalar and AVX frequencies across processor's cores. **Note:** Runtime increases with core count. |
113
-
| memory | runs [Intel(r) Memory Latency Checker](https://www.intel.com/content/www/us/en/download/736633/intel-memory-latency-checker-intel-mlc.html) (MLC) to measure memory bandwidth and latency across a load range. **Note: MLC is not included with PerfSpect.** It can be downloaded from [here](https://www.intel.com/content/www/us/en/download/736633/intel-memory-latency-checker-intel-mlc.html). Once downloaded, extract the Linux executable and place it in the perfspect/tools/x86_64 directory. |
129
+
| frequency | runs [avx-turbo](https://github.com/travisdowns/avx-turbo) to measure scalar and AVX frequencies across processor's cores. **Note:** Runtime increases with core count. |
130
+
| memory | runs [Intel(r) Memory Latency Checker](https://www.intel.com/content/www/us/en/download/736633/intel-memory-latency-checker-intel-mlc.html) (MLC) to measure memory bandwidth and latency across a load range. **Note: MLC is not included with PerfSpect.** It can be downloaded from [Intel Memory Latency Checker](https://www.intel.com/content/www/us/en/download/736633/intel-memory-latency-checker-intel-mlc.html). Once downloaded, extract the Linux executable and place it in the perfspect/tools/x86_64 directory. |
114
131
| numa | runs Intel(r) Memory Latency Checker(MLC) to measure bandwidth between NUMA nodes. See Note above about downloading MLC. |
115
132
| storage | runs [fio](https://github.com/axboe/fio) for 2 minutes across multiple I/O patterns to measure storage latency, IOPs, and bandwidth. Use --storage-dir to override the default location (/tmp). Minimum 32GB disk space required. |
116
133
117
134
#### Telemetry Command
135
+
118
136
The `telemetry` command reports CPU utilization, instruction mix, disk stats, network stats, and more on the specified target(s). All telemetry types are collected by default. To choose telemetry types, see the additional command line options (`perfspect telemetry -h`).
119
137
120
138

121
139
122
140
#### Flamegraph Command
141
+
123
142
Software flamegraphs are useful in diagnosing software performance bottlenecks. Run `perfspect flamegraph` to capture a system-wide software flamegraph.
124
143
144
+
> [!TIP]
145
+
> By default, flamegraphs are collected using the `cycles:P` event. To analyze different performance aspects, use the `--perf-event` flag to specify an alternative perf event (e.g., `cache-misses`, `instructions`, `branches`, `context-switches`, `mem-loads`, `mem-stores`, etc.).
146
+
125
147

126
148
127
149
#### Lock Command
150
+
128
151
As systems contain more and more cores, it can be useful to analyze the Linux kernel lock overhead and potential false-sharing that impacts system scalability. Run `perfspect lock` to collect system-wide hot spot, cache-to-cache and lock contention information. Experienced performance engineers can analyze the collected information to identify bottlenecks.
129
152
130
153
#### Config Command
131
-
The `config` command provides a method to view and change various system configuration parameters. Run `perfspect config -h` to view the parameters that can be modified.
154
+
155
+
The `config` command provides a method to view and change various system configuration parameters. Run `perfspect config -h` to view the parameters that can be modified.
132
156
133
157
> [!WARNING]
134
158
> Misconfiguring the system may cause it to stop functionining. In some cases, a reboot may be required to restore default settings.
The current configuration can, optionally, be saved to a file using the `--record` flag. This creates a human-readable configuration file that can be used to restore settings later.
144
170
145
171
Example:
146
-
<pre>
172
+
173
+
```text
147
174
$ ./perfspect config --tdp 300 --record
148
175
Configuration recorded to: perfspect_2025-12-01_14-30-45/gnr_config.txt
149
-
</pre>
176
+
```
150
177
151
178
##### Restoring Configuration
179
+
152
180
The `config restore` subcommand restores configuration from a previously recorded file. This is useful for reverting changes or applying a known-good configuration across multiple systems.
> The restore command works with remote targets too. Use `--target` or `--targets` to restore configuration on remote systems.
173
203
174
204
### Common Command Options
175
205
176
206
#### Local vs. Remote Targets
207
+
177
208
By default, PerfSpect targets the local host, that is, the host where PerfSpect is running. Remote systems can also be targeted if they are reachable via SSH from the local host.
178
209
179
210
> [!IMPORTANT]
180
211
> Ensure the remote user has password-less sudo access (or root privileges) to fully utilize PerfSpect's capabilities.
181
212
182
213
To target a single remote system with a pre-configured private key:
183
-
<pre>
184
-
$ ./perfspect report --target 192.168.1.42 --user fred --key ~/.ssh/fredkey
185
-
...
186
-
</pre>
214
+
215
+
```text
216
+
./perfspect report --target 192.168.1.42 --user fred --key ~/.ssh/fredkey
217
+
```
218
+
187
219
To target a single remote system with a password:
188
-
<pre>
189
-
$ ./perfspect report --target 192.168.1.42 --user fred
220
+
221
+
```text
222
+
./perfspect report --target 192.168.1.42 --user fred
To target more than one remote system, a YAML file with the necessary connection parameters is provided to PerfSpect. Refer to the example YAML file: [targets.yaml](targets.yaml).
194
-
<pre>
195
-
$ ./perfspect report --targets mytargets.yaml
196
-
...
197
-
</pre>
227
+
228
+
```text
229
+
./perfspect report --targets mytargets.yaml
230
+
```
198
231
199
232
> [!NOTE]
200
233
> All PerfSpect commands support remote targets, but some command options are limited to the local target.
201
234
202
235
#### Output
236
+
203
237
##### Logging
238
+
204
239
By default, PerfSpect writes to a log file (perfspect.log) in the user's current working directory. Optionally, PerfSpect can direct logs to the local system's syslog daemon.
205
-
<pre>
206
-
$ ./perfspect metrics --syslog
207
-
</pre>
240
+
241
+
```text
242
+
./perfspect metrics --syslog
243
+
```
208
244
209
245
##### Report Files
246
+
210
247
By default, PerfSpect creates a unique directory in the user's current working directory to store output files. Users can specify a custom output directory with the --output flag.
> Skip the build. Pre-built PerfSpect releases are available in the repository's [Releases](https://github.com/intel/PerfSpect/releases). Download and extract perfspect.tgz.
257
+
218
258
### 1st Build
259
+
219
260
Use `builder/build.sh` to build the dependencies and the application in Docker containers with the required build environments. Ensure Docker is properly configured on your build system before running the script.
220
261
221
262
### Subsequent Builds
263
+
222
264
`make` builds the app. It assumes the dependencies have been built previously and that you have Go installed on your development system. See [go.mod](go.mod) for the minimum Go version.
0 commit comments