docs: comprehensive documentation accuracy audit and fixes

[ajcasagrande]

Summary

• Systematic audit of 58 documentation files against the actual codebase
• Fixed 63 issues across core docs, tutorials, API references, and diagrams
• Used markdown-accuracy-auditor agent to verify all code examples, class names, CLI options, and file paths

Key Fixes

Core Documentation:

• Fix broken link in README (trace-benchmarking.md → trace_replay.md)
• Update architecture.md with GPU Telemetry and Server Metrics Manager services
• Fix copyright dates to 2025-2026

CLI & Configuration:

• Fix cli_options.md typos and missing constraint annotations
• Fix patterns.md missing MessageType import and mock_plugin example
• Add missing chat_embeddings endpoint and dataset loaders to plugin-system.md

Metrics & Data:

• Fix metrics_reference.md ITL requirements and formula corrections
• Add missing synthetic video, multi_turn, random_pool to benchmark_datasets.md
• Fix tutorial.md output format and artifact path

Tutorials (15 files):

• Fix CLI option names (--output-dir → --artifact-dir)
• Fix endpoint paths (/v1/rerank → /v2/rerank)
• Fix file extensions (.json → .jsonl)
• Fix field names, default values, and code block formatting

API & Architecture:

• Fix synthesis.md RadixTreeStats format and add missing SynthesisParams field
• Add missing RNG identifiers to reproducibility.md
• Fix metrics-flow.md class names (TotalRequestsMetric → RequestCountMetric)
• Fix mixins.md inheritance diagram with missing intermediate classes
• Fix CONTRIBUTING.md formatter reference (black → ruff)

Test plan

• All pre-commit hooks pass
• Plugin schema validation passes
• CLI docs auto-generation passes
• Visual review of key documentation pages

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

• New Features

  • Added chat embeddings and Cohere rankings endpoint support
  • Introduced custom dataset loaders for multi-turn, single-turn, mooncake trace, and random pool configurations
  • Added concurrency_burst arrival pattern option

• Updates

  • Updated CLI flags: --request-rate-mode → --arrival-pattern and --output-dir → --artifact-dir
  • Fixed schedule format now uses text_input field instead of input_text
  • GPU telemetry export format changed to JSON Lines format
  • Expanded default visualization plots for performance analysis

• Documentation

  • Comprehensive architecture updates for GPU Telemetry and Server Metrics management
  • Enhanced benchmark datasets guidance with new dataset type examples
  • Improved metrics reference and reproducibility documentation

[github-actions]

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@84d341e4b98419818e0d1b6d5f116aeb7ced0cb1

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@84d341e4b98419818e0d1b6d5f116aeb7ced0cb1

---

Last updated for commit: 84d341e • Browse code

[coderabbitai]

Walkthrough

This pull request updates documentation across the project and includes a minor configuration description change. The updates cover architecture documentation for new telemetry managers, tutorial and plugin documentation for new features (dataset loaders, endpoints, arrival patterns), metrics and timing reference updates, and various CLI flag/example corrections throughout tutorials and guidelines.

Changes

Cohort / File(s)	Summary
Contributing Guidelines CONTRIBUTING.md, README.md	Updated documentation links and formatter examples (black → ruff), adjusted trace benchmarking documentation reference.
Architecture & Diagrams docs/architecture.md, docs/diagrams/mixins.md, docs/diagrams/metrics-flow.md	Expanded architecture documentation with GPU Telemetry Manager and Server Metrics Manager sections; introduced new mixin classes (CommunicationMixin, CommandHandlerMixin); updated metric labels in flow diagrams.
API & Plugin Documentation docs/api/synthesis.md, docs/plugins/plugin-system.md	Added renormalize_hash_ids configuration option to synthesis API; introduced new endpoint types (chat_embeddings, cohere_rankings, nim_embeddings) and custom dataset loaders (mooncake_trace, multi_turn, random_pool, single_turn).
Benchmark & Timing Reference docs/benchmark_datasets.md, docs/benchmark_modes/timing-modes-reference.md, docs/benchmark_modes/trace_replay.md	Expanded benchmark dataset documentation with synthetic video and custom dataset types; refined timing-modes prerequisites and validation ranges; clarified trace replay field requirements per scheduling mode.
Metrics & Telemetry Reference docs/metrics_reference.md, docs/server_metrics/server_metrics_reference.md, docs/reproducibility.md, docs/genai-perf-feature-comparison.md	Updated ITL requirement from chunk count to token-length condition; added new RNG identifier strings; renamed histogram field (stats.total → stats.count); added new endpoint type support comparison rows.
Tutorials docs/tutorial.md, docs/tutorials/arrival-patterns.md, docs/tutorials/fixed-schedule.md, docs/tutorials/gpu-telemetry.md, docs/tutorials/multi-turn.md, docs/tutorials/multi-url-load-balancing.md, docs/tutorials/plot.md, docs/tutorials/prefix-synthesis.md, docs/tutorials/rankings.md, docs/tutorials/request-cancellation.md, docs/tutorials/sglang-image-generation.md, docs/tutorials/synthetic-video.md, docs/tutorials/template-endpoint.md, docs/tutorials/timeslices.md	Updated CLI flags (--output-dir → --artifact-dir, --request-rate-mode → --arrival-pattern), changed input field names (input_text → text_input), added concurrency_burst arrival pattern, updated file formats and example outputs, expanded plot documentation with multi-run and dashboard examples.
Development Documentation docs/dev/patterns.md	Updated code sample imports and mock plugin testing pattern in documentation examples.
Source Configuration src/aiperf/timing/config.py	Updated CreditPhaseConfig.request_rate_ramp_duration_sec description to clarify ramp starting point.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Poem

🐰 Through docs we hop, from start to end,
New features bloom, old paths we mend,
From telemetry's gleam to dataset's dance,
With clearer flags and tuning's chance,
A burrow of updates, thoughtfully penned! ✨

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: a comprehensive documentation accuracy audit and fixes across multiple documentation files.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

docs/tutorials/arrival-patterns.md (1)

112-120: ⚠️ Potential issue | 🟡 Minor

Remove incorrect constraint on line 361: --arrival-pattern does not universally require --request-rate.

The concurrency_burst pattern explicitly requires --request-rate to be omitted. Line 361 states "--arrival-pattern requires --request-rate to be set", which is factually incorrect and contradicts both the tutorial section (lines 112–120) and the code behavior. Update the constraint to clarify that concurrency_burst works without --request-rate, while constant, poisson, and gamma require it.

🤖 Fix all issues with AI agents

In `@docs/tutorials/plot.md`:
- Around line 27-39: The fenced code block under the "**Sample Output
(Successful Run):**" section lacks a language tag; update the opening fence from
``` to ```text (or ```console) so Markdownlint recognizes it as a code/console
block and the example output renders correctly.

In `@docs/tutorials/timeslices.md`:
- Around line 111-112: The docs list of Stats includes "std" but the examples
don't; update the timeslices documentation by either adding "std" values to the
CSV and JSON examples used in the Timeslices tutorial samples (the CSV sample
block and the JSON sample block) so they include a std column/field formatted
like other numeric stats (2 decimal places), or remove "std" from the Stat
description if std is not actually exported—ensure the CSV sample headers and
rows and the JSON sample objects are consistent with the chosen approach.

🧹 Nitpick comments (1)

docs/benchmark_datasets.md (1)

36-40: Verify FFmpeg availability documentation.

The Synthetic Video entry correctly notes that FFmpeg is required. However, users may need guidance on:

1. How to install FFmpeg
2. What happens if FFmpeg is missing (error message, graceful degradation)
3. Which video formats are supported without FFmpeg, if any

Consider adding a link to installation instructions or a troubleshooting section for FFmpeg-related issues.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Add a language to the fenced code block.

Markdownlint flags the sample output block because the fence lacks a language tag. Add text (or console) to the opening fence.

Suggested fix

-```
+```text
 INFO     Loading single-run data from: artifacts/Qwen_Qwen3-0.6B-chat-concurrency10/
 INFO     Detected mode: SINGLE_RUN
 INFO     Generating 5 time series plots
 INFO     Creating plot: ttft_over_time.png
 INFO     Creating plot: ttft_timeline.png
 INFO     Creating plot: timeslices_ttft.png
 INFO     Creating plot: timeslices_itl.png
 INFO     Creating plot: gpu_utilization_and_throughput_over_time.png
 INFO     Successfully generated 5 plots
 INFO     Plots saved to: artifacts/Qwen_Qwen3-0.6B-chat-concurrency10/plots/
-```
+```

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	**Sample Output (Successful Run):**
	```
	INFO Loading single-run data from: artifacts/Qwen_Qwen3-0.6B-chat-concurrency10/
	INFO Detected mode: SINGLE_RUN
	INFO Generating 4 time series plots
	INFO Generating 5 time series plots
	INFO Creating plot: ttft_over_time.png
	INFO Creating plot: itl_over_time.png
	INFO Creating plot: latency_over_time.png
	INFO Creating plot: dispersed_throughput_over_time.png
	INFO Successfully generated 4 plots
	INFO Creating plot: ttft_timeline.png
	INFO Creating plot: timeslices_ttft.png
	INFO Creating plot: timeslices_itl.png
	INFO Creating plot: gpu_utilization_and_throughput_over_time.png
	INFO Successfully generated 5 plots
	INFO Plots saved to: artifacts/Qwen_Qwen3-0.6B-chat-concurrency10/plots/
	```
	**Sample Output (Successful Run):**

🧰 Tools

🪛 markdownlint-cli2 (0.20.0)

[warning] 28-28: Fenced code blocks should have a language specified

(MD040, fenced-code-language)

🤖 Prompt for AI Agents

In `@docs/tutorials/plot.md` around lines 27 - 39, The fenced code block under the
"**Sample Output (Successful Run):**" section lacks a language tag; update the
opening fence from ``` to ```text (or ```console) so Markdownlint recognizes it
as a code/console block and the example output renders correctly.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Documentation inconsistency: std added to description but not shown in examples.

The Stat description now includes std, but the CSV sample (lines 91-105) and JSON sample (lines 121-163) don't include std values. Consider adding std entries to the examples for consistency, or verify whether std is actually exported in timeslice data.

🤖 Prompt for AI Agents

In `@docs/tutorials/timeslices.md` around lines 111 - 112, The docs list of Stats
includes "std" but the examples don't; update the timeslices documentation by
either adding "std" values to the CSV and JSON examples used in the Timeslices
tutorial samples (the CSV sample block and the JSON sample block) so they
include a std column/field formatted like other numeric stats (2 decimal
places), or remove "std" from the Stat description if std is not actually
exported—ensure the CSV sample headers and rows and the JSON sample objects are
consistent with the chosen approach.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[debermudez]

are you adding the agent file as well?