Add an option to produce fine-grained HTML renderers based on CommonMark's grammar

[herley-shaori]

Summary

Implements the first task of #606 by exposing CommonMark's HTML block type differentiation through individual renderers.

A new boolean option parseHtmlBlocks (default: false) is added. When enabled alongside the html option, the parser produces type-specific renderers instead of the generic inputBlockHtmlElement and inlineHtmlTag renderers:

Block HTML renderers (each receives a filename of a file containing the HTML block contents):

Renderer	CommonMark type	Matches
inputBlockHtmlCommentElement	Type 2	<!-- ... -->
inputBlockHtmlInstructionElement	Type 3	<? ... ?>
inputBlockHtmlDeclarationElement	Type 4	<! ... >
inputBlockHtmlCdataElement	Type 5	<![CDATA[ ... ]]>
inputBlockHtmlSpecialElement	Type 1	<script>, <pre>, <style>, <textarea>
inputBlockHtmlRegularElement	Type 6	<div>, <table>, <form>, etc.
inputBlockHtmlAnyElement	Type 7	Any other complete tag on its own line

Inline HTML renderers (each receives the tag contents as a string):

Renderer	Matches
inlineHtmlInstruction	<? ... ?>
inlineHtmlDeclaration	<! ... >
inlineHtmlCdataSection	<![CDATA[ ... ]]>
inlineHtmlOpenTag	<tag>
inlineHtmlCloseTag	</tag>
inlineHtmlEmptyTag	<tag/>

The existing inlineHtmlComment renderer remains unchanged. When parseHtmlBlocks is false (default), behavior is fully backward compatible.

Changes

• markdown.dtx: Added parseHtmlBlocks option definition, 13 new renderer registrations, 13 new Lua writer functions, conditional parser routing in DisplayHtml and InlineHtml, and documentation for all new renderers.
• tests/support/keyval-setup.tex: Added test renderer prototypes for all 13 new renderers.
• tests/testfiles/regression/github/issue-606-block-html-types.test: New regression test verifying type-specific renderers are produced when parseHtmlBlocks is enabled.

Test plan

• New regression test passes across all non-ConTeXt TeX formats (luatex, lualatex, pdflatex, pdftex)
• All 44 existing CommonMark_0.30/html_blocks tests pass (backward compatibility)
• Both existing CommonMark_0.31.2/raw_html tests pass (backward compatibility)
• ConTeXt tests (skipped locally due to missing rename utility — CI should cover this)

Note

This PR addresses Task 1 of #606. Task 2 (renderers corresponding to HTML nodes) would require a more substantial HTML parser and is left for a follow-up.

Continues #606.

[Witiko]

Hi @herley-shaori, this looks great, especially for a first-time contribution. Thanks for putting in the effort!

Below, I reviewed the code. If you'd like to make the necessary changes yourself, that would be great; otherwise, I'm happy to take over the PR from here.

[Witiko]

Tasks for myself in addition to the comments from the code review:

• Set htmlOutput = "commonmark" when the experimental option has been enabled.
• Improve the naming of writers, i.e. why block_html_comment_element when it's an HTML node (not element).
  • Same for renderers and renderer prototypes, why \markdownRendererInputBlockHtmlCommentElement rather than just \markdownRendererInputBlockHtmlComment?
• Add a code example redefining \markdownRendererInputBlockHtmlComment to the user manual, similar to the existing code example for \markdownRendererInlineComment.
  • Add other code examples, e.g. redefining \markdownRendererInlineHtmlOpenTag and \markdownRendererInlineHtmlCloseTag, as requested by @u-fischer in handling inline html #597 (comment).
• Update CHANGES.md.

[herley-shaori]

Hi @Witiko, thanks so much for taking the time to review this! Your feedback is really valuable. I realize I still have a lot to learn — I'll step back and leave this PR to you. If there's anything I can do to help in the future, happy to contribute! 😊

[Witiko]

Everything seems OK. I am waiting with the merge until after the v3.14.1 bugfix release, likely at the end of this week, so that we don't ship new features with it.

[Witiko]

@herley-shaori: I found a few other issues that still need to be addressed. If you have the time and would like to, please feel free to take care of them; otherwise, I can finish them myself.

[Witiko]

I don't think this is currently implemented, i.e. the prototypes produce an empty expansion at the moment.

Default definitions that fall back onto \markdownRendererInputBlockHtmlElementPrototype should be added to the section ### Token Renderer Prototypes {#tex-token-renderer-prototypes}.

Here is an easy way to verify that this was implemented correctly: after this change, the test file no-html-output.test should pass even when htmlOutput = commonmark is added to the \markdownSetup command at the top of the file and all newly added definitions are removed from keyval-setup.tex.

[Witiko]

Furthermore, blockHtmlStandaloneTag can't really fall back onto the inputBlockHtmlElement due to the argument mismatch. Instead, it may need to fall back onto e.g. the inlineHtmlTag prototype.

[Witiko]

I don't think this is currently implemented, i.e. the prototypes produce an empty expansion at the moment.

Default definitions that fall back onto \markdownRendererInlineHtmlTagPrototype should be added to the section ### Token Renderer Prototypes {#tex-token-renderer-prototypes}.

Here is an easy way to verify that this was implemented correctly: after this change, the test file no-html-output.test should pass even when htmlOutput = commonmark is added to the \markdownSetup command at the top of the file and all newly added definitions are removed from keyval-setup.tex.

[Witiko]

This still won't work, since we don't strip the leading <!-- and the trailing -->. For consistency with the inlineHtmlComment renderer, we should likely strip these, parse the content and provide it directly, not through a separate file, falling back on the inlineHtmlComment renderer prototype instead of inputBlockHtmlElement.

Then, this example should be updated as follows:

Suggested change

	inputBlockHtmlComment = {\marginnote{\input{#1}}},
	blockHtmlComment = \marginnote{#1},