Skip to content

Commit 08d5307

Browse files
chrisjsewellchrisjsewell
authored
Merge branch 'master' into better-meta
2 parents cf26dea + e0fc7a3 commit 08d5307

File tree

3 files changed

+363
-0
lines changed

3 files changed

+363
-0
lines changed

‎AGENTS.md‎

Lines changed: 341 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,341 @@
1+
# AGENTS.md
2+
3+
This file provides guidance for AI coding agents working on the **MyST-Parser** repository.
4+
5+
## Project Overview
6+
7+
MyST-Parser is a Sphinx extension and docutils parser for the MyST (Markedly Structured Text) Markdown flavor. It provides:
8+
9+
- An extended [CommonMark](https://commonmark.org)-compliant parser using [`markdown-it-py`](https://markdown-it-py.readthedocs.io/)
10+
- A [docutils](https://docutils.sourceforge.io/) renderer that converts markdown-it tokens to docutils nodes
11+
- A [Sphinx](https://www.sphinx-doc.org) extension for using MyST Markdown in Sphinx documentation
12+
13+
MyST is designed for technical documentation and publishing, offering a rich and extensible flavor of Markdown with support for roles, directives, and cross-references.
14+
15+
## Repository Structure
16+
17+
```
18+
pyproject.toml # Project configuration and dependencies
19+
tox.ini # Tox test environment configuration
20+
21+
myst_parser/ # Main source code
22+
├── __init__.py # Package init with Sphinx setup() entry point
23+
├── config/ # Configuration dataclasses
24+
│ ├── main.py # MdParserConfig dataclass
25+
│ └── dc_validators.py # Dataclass field validators
26+
├── parsers/ # Parser implementations
27+
│ ├── sphinx_.py # Sphinx parser (MystParser)
28+
│ ├── docutils_.py # Docutils parser and CLI tools
29+
│ ├── mdit.py # markdown-it-py setup and plugins
30+
│ ├── directives.py # Directive parsing utilities
31+
│ └── options.py # Option parsing for directives
32+
├── mdit_to_docutils/ # Token-to-docutils rendering
33+
│ ├── base.py # DocutilsRenderer (main renderer)
34+
│ ├── sphinx_.py # SphinxRenderer (Sphinx-specific)
35+
│ ├── transforms.py # Docutils transforms
36+
│ └── html_to_nodes.py # HTML-to-docutils conversion
37+
├── sphinx_ext/ # Sphinx extension components
38+
│ ├── main.py # setup_sphinx() and config creation
39+
│ ├── directives.py # Custom Sphinx directives
40+
│ ├── myst_refs.py # Reference resolver post-transform
41+
│ └── mathjax.py # MathJax configuration
42+
├── inventory.py # Sphinx inventory file handling
43+
├── mocking.py # Mock objects for directive/role parsing
44+
├── warnings_.py # Warning types (MystWarnings enum)
45+
├── cli.py # Command-line interface
46+
└── _compat.py # Python version compatibility
47+
48+
tests/ # Test suite
49+
├── test_sphinx/ # Sphinx integration tests
50+
│ ├── sourcedirs/ # Test documentation projects
51+
│ └── test_sphinx_builds.py
52+
├── test_renderers/ # Renderer unit tests
53+
├── test_commonmark/ # CommonMark compliance tests
54+
├── test_html/ # HTML output tests
55+
├── test_docutils.py # Docutils parser tests
56+
└── test_anchors.py # Heading anchor tests
57+
58+
docs/ # Documentation source (MyST Markdown)
59+
├── conf.py # Sphinx configuration
60+
├── index.md # Documentation index
61+
├── syntax/ # Syntax reference documentation
62+
├── develop/ # Developer documentation
63+
└── faq/ # FAQ and troubleshooting
64+
```
65+
66+
## Development Commands
67+
68+
All commands should be run via [`tox`](https://tox.wiki) for consistency. The project uses `tox-uv` for faster environment creation.
69+
70+
### Testing
71+
72+
```bash
73+
# Run all tests
74+
tox
75+
76+
# Run a specific test file
77+
tox -- tests/test_docutils.py
78+
79+
# Run a specific test function
80+
tox -- tests/test_docutils.py::test_function_name
81+
82+
# Run tests with a specific Python/Sphinx version
83+
tox -e py311-sphinx8
84+
85+
# Run with coverage
86+
tox -- --cov=myst_parser
87+
88+
# Update regression test fixtures
89+
tox -- --regen-file-failure --force-regen
90+
```
91+
92+
### Documentation
93+
94+
```bash
95+
# Build docs (clean)
96+
tox -e docs-clean
97+
98+
# Build docs (incremental)
99+
tox -e docs-update
100+
101+
# Build with a specific builder (e.g., linkcheck)
102+
BUILDER=linkcheck tox -e docs-update
103+
```
104+
105+
### Code Quality
106+
107+
```bash
108+
# Type checking with mypy
109+
tox -e mypy
110+
111+
# Linting with ruff (auto-fix enabled)
112+
tox -e ruff-check
113+
114+
# Formatting with ruff
115+
tox -e ruff-fmt
116+
117+
# Run pre-commit hooks on all files
118+
pre-commit run --all-files
119+
```
120+
121+
## Code Style Guidelines
122+
123+
- **Formatter/Linter**: Ruff (configured in `pyproject.toml`)
124+
- **Type Checking**: Mypy with strict settings (configured in `pyproject.toml`)
125+
- **Pre-commit**: Use pre-commit hooks for consistent code style
126+
127+
### Best Practices
128+
129+
- **Type annotations**: Use complete type annotations for all function signatures. Use `TypedDict` for structured dictionaries, dataclasses for configuration.
130+
- **Docstrings**: Use Sphinx-style docstrings (`:param:`, `:return:`, `:raises:`). Types are not required in docstrings as they should be in type hints.
131+
- **Function Signatures**: Use `/` and `*` to enforce positional-only and keyword-only arguments where appropriate
132+
- **Pure functions**: Where possible, write pure functions without side effects.
133+
- **Error handling**: Use `MystWarnings` enum for warning types. Use `create_warning()` for user-facing warnings.
134+
- **Testing**: Write tests for all new functionality. Use `pytest-regressions` for output comparison tests.
135+
136+
### Docstring Example
137+
138+
```python
139+
def parse_directive_text(
140+
directive_class: type[Directive],
141+
first_line: str,
142+
content: str,
143+
*,
144+
validate_options: bool = True,
145+
) -> tuple[list[str], dict[str, Any], list[str], int]:
146+
"""Parse directive text into its components.
147+
148+
:param directive_class: The directive class to parse for.
149+
:param first_line: The first line (arguments).
150+
:param content: The directive content.
151+
:param validate_options: Whether to validate options against the directive spec.
152+
:return: Tuple of (arguments, options, body_lines, body_offset).
153+
:raises MarkupError: If the directive text is malformed.
154+
"""
155+
...
156+
```
157+
158+
## Testing Guidelines
159+
160+
### Test Structure
161+
162+
- Tests use `pytest` with fixtures from `conftest.py` files
163+
- Sphinx integration tests are in `tests/test_sphinx/`
164+
- Test source directories are in `tests/test_sphinx/sourcedirs/`
165+
- Regression testing uses `pytest-regressions` for output comparison
166+
- Use `pytest-param-files` for parameterized file-based tests
167+
168+
### Writing Tests
169+
170+
1. For Sphinx integration tests, create a source directory in `tests/test_sphinx/sourcedirs/`
171+
2. Use `sphinx-pytest` fixtures for Sphinx application testing
172+
3. Use `file_regression` fixture for comparing output against stored fixtures
173+
174+
### Test Best Practices
175+
176+
- **Test coverage**: Write tests for all new functionality and bug fixes
177+
- **Isolation**: Each test should be independent and not rely on state from other tests
178+
- **Descriptive names**: Test function names should describe what is being tested
179+
- **Regression testing**: Use `file_regression.check()` for complex output comparisons
180+
- **Parametrization**: Use `@pytest.mark.parametrize` for multiple test scenarios
181+
- **Fixtures**: Define reusable fixtures in `conftest.py`
182+
183+
### Example Test Pattern
184+
185+
```python
186+
import pytest
187+
188+
@pytest.mark.sphinx(
189+
buildername="html",
190+
srcdir="path/to/sourcedir",
191+
)
192+
def test_example(app, status, warning, get_sphinx_app_output):
193+
app.build()
194+
assert "build succeeded" in status.getvalue()
195+
warnings = warning.getvalue().strip()
196+
assert warnings == ""
197+
```
198+
199+
## Pull Request Requirements
200+
201+
When submitting changes:
202+
203+
1. **Description**: Include a meaningful description or link explaining the change
204+
2. **Tests**: Include test cases for new functionality or bug fixes
205+
3. **Documentation**: Update docs if behavior changes or new features are added
206+
4. **Changelog**: Update `CHANGELOG.md` under the appropriate section
207+
5. **Code Quality**: Ensure `pre-commit run --all-files` passes
208+
209+
## Architecture Overview
210+
211+
### Parsing Pipeline
212+
213+
The MyST parser follows a three-stage pipeline:
214+
215+
```
216+
MyST Markdown → markdown-it tokens → docutils AST → Sphinx/HTML output
217+
```
218+
219+
1. **Markdown Parsing** (`myst_parser/parsers/mdit.py`): Uses `markdown-it-py` with MyST plugins to parse Markdown into tokens
220+
2. **Token Rendering** (`myst_parser/mdit_to_docutils/`): Converts markdown-it tokens to docutils nodes
221+
3. **Sphinx Integration** (`myst_parser/sphinx_ext/`): Integrates with Sphinx for documentation builds
222+
223+
### Key Components
224+
225+
#### Configuration (`myst_parser/config/main.py`)
226+
227+
The `MdParserConfig` dataclass centralizes all MyST configuration options:
228+
229+
- Defines all `myst_*` configuration values with types and defaults
230+
- Uses dataclass validators for type checking
231+
- Sphinx config values are auto-registered with `myst_` prefix
232+
233+
#### Parsers (`myst_parser/parsers/`)
234+
235+
- `MystParser` (in `sphinx_.py`): The Sphinx parser class that integrates with Sphinx's build system
236+
- `Parser` (in `docutils_.py`): The standalone docutils parser for non-Sphinx use
237+
- `create_md_parser()` (in `mdit.py`): Factory function to create configured markdown-it-py instances
238+
239+
#### Renderers (`myst_parser/mdit_to_docutils/`)
240+
241+
- `DocutilsRenderer` (in `base.py`): Base renderer that converts tokens to docutils nodes. Contains render methods for each token type (e.g., `render_heading`, `render_paragraph`).
242+
- `SphinxRenderer` (in `sphinx_.py`): Extends `DocutilsRenderer` with Sphinx-specific functionality (e.g., cross-references, domains).
243+
244+
#### Sphinx Extension (`myst_parser/sphinx_ext/`)
245+
246+
- `setup_sphinx()` (in `main.py`): Registers the parser, config values, and transforms with Sphinx
247+
- `MystReferenceResolver` (in `myst_refs.py`): Post-transform that resolves MyST-style references
248+
- Custom directives like `figure-md` (in `directives.py`)
249+
250+
### Sphinx Integration Flow
251+
252+
```mermaid
253+
flowchart TB
254+
subgraph init["Initialization"]
255+
setup["setup() in __init__.py"]
256+
setup_sphinx["setup_sphinx()"]
257+
end
258+
259+
subgraph config["Configuration"]
260+
builder_init["builder-inited"]
261+
create_config["create_myst_config()"]
262+
end
263+
264+
subgraph parse["Parse Phase"]
265+
source["Source .md file"]
266+
mdit["markdown-it-py parser"]
267+
tokens["Token stream"]
268+
renderer["DocutilsRenderer/SphinxRenderer"]
269+
doctree["Docutils doctree"]
270+
end
271+
272+
subgraph resolve["Resolution Phase"]
273+
post_transform["MystReferenceResolver"]
274+
resolved["Resolved doctree"]
275+
end
276+
277+
setup --> setup_sphinx
278+
setup_sphinx --> builder_init
279+
builder_init --> create_config
280+
281+
source --> mdit --> tokens --> renderer --> doctree
282+
doctree --> post_transform --> resolved
283+
284+
create_config -.->|"MdParserConfig"| renderer
285+
286+
style create_config fill:#e1f5fe
287+
style renderer fill:#e1f5fe
288+
style post_transform fill:#e1f5fe
289+
```
290+
291+
## Key Files
292+
293+
- `pyproject.toml` - Project configuration, dependencies, and tool settings
294+
- `myst_parser/__init__.py` - Package entry point with `setup()` for Sphinx
295+
- `myst_parser/config/main.py` - `MdParserConfig` dataclass with all configuration options
296+
- `myst_parser/parsers/sphinx_.py` - `MystParser` class for Sphinx integration
297+
- `myst_parser/mdit_to_docutils/base.py` - `DocutilsRenderer` with token-to-node rendering
298+
- `myst_parser/sphinx_ext/main.py` - `setup_sphinx()` function for Sphinx setup
299+
- `myst_parser/warnings_.py` - `MystWarnings` enum for warning types
300+
301+
## Debugging
302+
303+
- Build docs with `-T` flag for full tracebacks: `tox -e docs-clean -- -T ...`
304+
- Use `myst-docutils-html` CLI for standalone parsing without Sphinx
305+
- Check `docs/_build/` for build outputs
306+
- Use `tox -- -v --tb=long` for verbose test output with full tracebacks
307+
- Set `myst_debug_match = True` in Sphinx config to log matched syntax
308+
309+
## Common Patterns
310+
311+
### Adding a New Syntax Extension
312+
313+
1. Create a markdown-it-py plugin or use an existing one from `mdit-py-plugins`
314+
2. Register it in `myst_parser/parsers/mdit.py`
315+
3. Add configuration option in `MdParserConfig` if needed
316+
4. Add render method in `DocutilsRenderer` (e.g., `render_my_syntax`)
317+
5. Document in `docs/syntax/`
318+
6. Add tests in `tests/`
319+
320+
### Adding a Configuration Option
321+
322+
1. Add field to `MdParserConfig` in `myst_parser/config/main.py`
323+
2. Add validator if needed in `myst_parser/config/dc_validators.py`
324+
3. Document in `docs/configuration.md`
325+
4. Add tests for the new option
326+
327+
### Adding a Sphinx Directive
328+
329+
1. Create directive class in `myst_parser/sphinx_ext/directives.py`
330+
2. Register in `setup_sphinx()` in `myst_parser/sphinx_ext/main.py`
331+
3. Document in appropriate docs section
332+
4. Add tests in `tests/test_sphinx/`
333+
334+
## Reference Documentation
335+
336+
- [markdown-it-py Repository](https://github.com/ExecutableBookProject/markdown-it-py)
337+
- [markdown-it-py Documentation](https://markdown-it-py.readthedocs.io/)
338+
- [Docutils Repository](https://github.com/live-clones/docutils)
339+
- [Docutils Documentation](https://docutils.sourceforge.io/)
340+
- [Sphinx Repository](https://github.com/sphinx-doc/sphinx)
341+
- [Sphinx Extension Development](https://www.sphinx-doc.org/en/master/extdev/index.html)

‎pyproject.toml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,16 @@ testing-docutils = [
8383
"pytest-param-files~=0.6.0",
8484
]
8585

86+
[dependency-groups]
87+
mypy = [
88+
"mypy==1.19.1",
89+
"types-urllib3",
90+
"sphinx~=8.2",
91+
"markdown-it-py~=4.0",
92+
"mdit-py-plugins~=0.5.0",
93+
]
94+
ruff = ["ruff==0.14.11"]
95+
8696
[project.scripts]
8797
myst-anchors = "myst_parser.cli:print_anchors"
8898
myst-inv = "myst_parser.inventory:inventory_cli"

‎tox.ini‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,3 +48,15 @@ commands =
4848
--re-ignore _build/.* \
4949
--port 0 --open-browser \
5050
-n -b {posargs:html} docs/ docs/_build/{posargs:html}
51+
52+
[testenv:ruff-check]
53+
dependency_groups = ruff
54+
commands = ruff check {posargs:--fix}
55+
56+
[testenv:ruff-fmt]
57+
dependency_groups = ruff
58+
commands = ruff format {posargs}
59+
60+
[testenv:mypy]
61+
dependency_groups = mypy
62+
commands = mypy {posargs:myst_parser}

0 commit comments

Comments
 (0)