Skip to content

Conversation

@cscheid
Copy link
Collaborator

@cscheid cscheid commented Aug 20, 2025

Brings:

  • Pandoc to 3.8.3
  • typst to 0.14.2

@posit-snyk-bot
Copy link
Collaborator

posit-snyk-bot commented Aug 20, 2025

Snyk checks have passed. No issues have been found so far.

Status Scanner Critical High Medium Low Total (0)
Open Source Security 0 0 0 0 0 issues
Licenses 0 0 0 0 0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

@cscheid
Copy link
Collaborator Author

cscheid commented Aug 20, 2025

@tarleb It seems that there's a -f markdown change from Pandoc 3.6.3 to 3.7.0.2. Let's talk about it when we meet next? Thanks!

@tarleb
Copy link
Collaborator

tarleb commented Aug 21, 2025

Sorry, I hadn't seen this. I'll try to be there today.

@cscheid
Copy link
Collaborator Author

cscheid commented Aug 21, 2025

@tarleb suggests " -> \" as a path forward. Let's try it.

@cscheid
Copy link
Collaborator Author

cscheid commented Aug 21, 2025

@tarleb suggests " -> \" as a path forward. Let's try it.

On further thought, this only works if we know that the replacement for the shortcode is in a position where this escaping is required. In order to do that, we would have to enrich the LPEG parser to learn about parsing links and images, and then to process shortcodes differently inside links and images. That's a pretty large undertaking.

@cscheid
Copy link
Collaborator Author

cscheid commented Dec 13, 2025

@gordonwoodhull We should take a look at this PR together next week; there's some brand+typst failures on the update to Pandoc 3.8.3 that I think you'll be better equipped to sort out :)

Typst (sometimes +brand, often #NormalTok) failures:

JATS failures (looks like the Pandoc output doesn't validate?)

Docusaurus failures (apparently from display math markdown writer changing)

LaTeX compilation failures:

DocX snapshot failures:

@cscheid cscheid changed the title [deps] Pandoc 3.7.0.2 [deps] Pandoc 3.8.3, typst 0.14.2 Dec 13, 2025
@gordonwoodhull
Copy link
Contributor

I've pushed a fix: page inset needs to be typst raw inline.

Most of the other failures seem to be because we need to update the Typst templates from Pandoc to get Skylighting support. Not trivial as there's a bunch of new stuff, but not that bad.

@gordonwoodhull
Copy link
Contributor

gordonwoodhull commented Dec 17, 2025

Importing highlighting-definitions in the typst template fixes all of the typst failures except for this listings one:

docs/smoke-all/crossrefs/float/typst/typst-listings-1.qmd

So all the stuff I need to update (merging Chris Kenny's improvements with brand yaml, basically) is not reflected in tests.

TL;DR We've been applying highlight-style for all formats, it just wasn't implemented for Typst before.

Now syntax highlighting will match by default!

Claude's explanation:

Quarto always sets highlight-style for ALL formats, including typst:

  1. Default value: kDefaultHighlightStyle = "arrow" defined in src/command/render/constants.ts:8 since March 2021
  2. Applied unconditionally: resolveTextHighlightStyle() in src/command/render/pandoc.ts:1741 is called for all formats with no format-specific checks
  3. For typst: Since extras.html?.[kTextHighlightingMode] is undefined for typst, it falls through to the default case (line 1789-1796) which sets highlight-style to the light theme path
  4. Result: Quarto passes --highlight-style=<path-to-arrow-light.theme> to pandoc even for typst output

Before pandoc 3.8/3.9: This didn't matter because pandoc ignored the highlight-style for typst and used typst's native highlighting.

After pandoc 3.8/3.9: Pandoc now respects --syntax-highlighting for typst output, so quarto's default "arrow" style triggers skylighting for ALL code blocks, including unlanguaged ones (wrapped in NormalTok).

What is NormalTok and why are unlanguaged blocks also invoking skylighting?

NormalTok is a TokenType in skylighting for "regular, unhighlighted text within a highlighted code block."

The typst writer explicitly adds a "default" class to code blocks before calling highlight, even when no language is specified. This allows skylighting to find a syntax definition and process all code blocks.

So the flow is:

  1. Quarto passes --syntax-highlighting=arrow to pandoc (the default)
  2. Pandoc's typst writer adds "default" class to unlanguaged code blocks
  3. Skylighting finds a syntax for "default" and tokenizes everything as NormalTok
  4. Output: #Skylighting(([#NormalTok("some code.");],))
  5. This triggers highlighting-definitions to be included in the template

@cderv cderv linked an issue Dec 17, 2025 that may be closed by this pull request
@cderv
Copy link
Collaborator

cderv commented Dec 18, 2025

Thanks @gordonwoodhull! What I mentioned yesterday is also that Pandoc now uses --syntax-highlighting flag. This is new from pandoc 3.8 https://pandoc.org/releases.html#pandoc-3.8-2025-09-06

Add a new command line option --syntax-highlighting; this takes the values none, default, idiomatic, a style name, or a path to a theme file. It replaces the --no-highlighting, --highlighting-style, and --listings options, which will still work but with a deprecation warning. (Albert Krewinkel)

So I think we need to check if we use that flag, and probably anticipate the end of deprecation warning in our default file use.

We've been applying highlight-style for all formats,

The new option is now syntax-highlighting to use in the default file
https://pandoc.org/MANUAL.html#general-writer-options-1

Currently we use

highlight-style: C:\Users\chris\Documents\DEV_R\quarto-cli\src\resources\pandoc\highlight-styles\arrow-light.theme

And maybe in other places.

Should we do this in another PR maybe ? It is kind of part of the pandoc update though.

@cderv
Copy link
Collaborator

cderv commented Dec 19, 2025

I am just updating to benefit from new condensed logging

@cscheid
Copy link
Collaborator Author

cscheid commented Dec 19, 2025

I think the biggest cause of failures right now if from a LaTeX change in how table counters work in the Pandoc writer and templates. I still don't understand how to avoid that problem.

@cderv
Copy link
Collaborator

cderv commented Dec 19, 2025

Oh, I have dealt with that recently for the rticles R package. I needed to add the missing template part to the rticles' built-in template (rstudio/rticles#596)
Initially, this is the Pandoc change jgm/pandoc@e13aa5c following report jgm/pandoc#11141 and then the latex template change: jgm/pandoc@d835461 (jgm/pandoc#11201).

They do something in the Writer that expects something in the template. It could be either this template addition, or some adaptation in the way we create the table.

Also related, this change in Pandoc is for ensuring that unlabelled tables don't increment counter. I remember this was a problem in cross ref with counter workaround to decrement or something, so it could be now conflicting with what Pandoc is doing.

Hope it helps understand. (Disclaimer: I did not look at failing examples, just what I looked for my rticles maintainance about this)

gordonwoodhull and others added 9 commits December 30, 2025 11:21
The functions in engine-shared.ts (languagesInMarkdown, languagesInMarkdownFile,
postProcessRestorePreservedHtml) have identical implementations in
src/core/pandoc/pandoc-partition.ts and src/core/jupyter/preserve.ts.

Update imports to use the canonical locations and remove the duplicate file.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
for pandoc 3.8

next: investigate why this case enables syntax-highlighting
@gordonwoodhull
Copy link
Contributor

Pushed the typst template updates.

The unconditional disabling of table_attributes in 72452ee caused errors for formats like commonmark_x that don't support this extension. Pandoc 3.8.3+ validates all extension keys (both enabled and disabled) and rejects unsupported extensions.

Use pandoc.format.all_extensions() to check if table_attributes is supported by the format before attempting to disable it.
@gordonwoodhull
Copy link
Contributor

gordonwoodhull commented Jan 8, 2026

Pushed fixes and tests to address #13857 (comment)

We were still calling the Typst columns() function, which is obsolete. We missed this change in the Pandoc template in Dec 2024 and now we have #set page(columns) in page.typ, equivalent to upstream setting page in the article function.

Thanks @mcanouil for reporting this!

Align Quarto's article() function with Pandoc's October 2024 fix
(commit e01023c1f) for Typst 0.12+ two-column layout compatibility.

Changes:
- Replace columns(cols, doc) wrapper with plain doc return
  (page.typ already sets columns via set page(columns:))
- Change title block wrapper from block() to
  place(top, float: true, scope: "parent", clearance: 4mm)
  so title/authors/abstract span both columns
- Remove unused cols parameter from typst-show.typ

This fixes the error "page configuration is not allowed inside of
containers" that occurred when using landscape sections with
two-column layouts, since columns() creates a container that
prevents set page() directives inside it.

Add smoke tests for two-column typst layouts that verify:
- Page columns set via set page(columns:) not columns() function
- Title block wrapped with place(scope: "parent") for column spanning
- Integration with landscape sections
- Integration with table of contents
- Full title block (title, subtitle, authors, abstract, thanks)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
cderv added 4 commits January 9, 2026 16:06
Add tests.run.skip field to allow unconditional test skipping with custom message.
Skip docusaurus equations test until Pandoc fix for display math newlines is released (fixed in pandoc@8123be6).
Pandoc 3.8+ wraps uncaptioned longtables with `{\def\LTcaptype{none}...}`
to prevent counter increment. When Quarto adds its own caption, this
definition conflicts with the caption package causing:
"Package caption Error: No float type 'none' defined."

Strip the LTcaptype definition while keeping the harmless { } wrapper.
- Skip test that are not relevant due to other failures
- Add more verify steps to really check more than just noErrors
…oc fix

Same issue as equations.qmd - Pandoc 3.7+ removes newlines in display math blocks.
Fixed upstream in pandoc@8123be6, awaiting release.
gordonwoodhull added a commit that referenced this pull request Jan 11, 2026
The previous show rule used .map().join() which destroyed the list
structure, breaking PDF/UA-1 validation. The new approach uses a nested
show rule that modifies items in place while preserving the original
terms element structure.

Fix suggested by @mcanouil in:
#13249 (comment)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
@cderv
Copy link
Collaborator

cderv commented Jan 12, 2026

The last missing part took me time to understand as it was an issue thrown in JSS but it is related to highlight-style: none. So what I mentioned at #13249 (comment) needs more work.

I'll add some tests as this should have surfaced without extensions. Last thing before merging hopefully!

@cderv
Copy link
Collaborator

cderv commented Jan 12, 2026

To keep change in this PR small enough as this is a big one yet, I'll fix the root cause, and do the deprecation handling in another PR

Pandoc 3.8 changed optSyntaxHighlighting from HighlightMethod type to Text.
The old HighlightMethod FromJSON accepted null, but Text rejects it with:
"Aeson exception: Error in $: parsing Text failed, expected String, but encountered Null"

Pass "none" string instead of null when highlight-style: none is set.
gordonwoodhull added a commit that referenced this pull request Jan 13, 2026
The previous show rule used .map().join() which destroyed the list
structure, breaking PDF/UA-1 validation. The new approach uses a nested
show rule that modifies items in place while preserving the original
terms element structure.

Fix suggested by @mcanouil in:
#13249 (comment)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
@cderv cderv merged commit 8a9e78e into main Jan 14, 2026
3 of 10 checks passed
@cderv cderv deleted the deps/pandoc-2025-08 branch January 14, 2026 11:00
gordonwoodhull added a commit that referenced this pull request Jan 14, 2026
The previous show rule used .map().join() which destroyed the list
structure, breaking PDF/UA-1 validation. The new approach uses a nested
show rule that modifies items in place while preserving the original
terms element structure.

Fix suggested by @mcanouil in:
#13249 (comment)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

"colorlinks: true" results in failed rendering of pdf Typst: Improve syntax highlighting support

7 participants