Improvements to performance

[chrisjsewell]

This is related to #957, but I wanted to have a place to more generally document possible/ongoing performance improvements to sphinx-needs.

Here is the result from the current code (c73c44f)

sphinx-analysis --project needs --pyinstrument --tree --tree-filter 0.01

There are a few things here to note:

1. The print_need_nodes function takes up ~25% of the runtime, and a significant time is spent in the Inliner and OptionParser initialisation
2. process_needtables also takes up ~10% of the time, with paragraph node intialisation taking noticable time
3. The SphinxSmartOuotes transform takes a significant amount of time

I have proposed fixes to all of these...

[chrisjsewell]

Improvements:

• 👌 Performance: Memoize Inline parser for build_need #968 massively improves the print_need_nodes performance
• https://sourceforge.net/p/docutils/patches/206/ looks to improve the performance of the SmartQuotes transform (although it is still quite slow)

[chrisjsewell]

Perhaps it would be good to

(a) advise to just set https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-smartquotes to False, and/or

(b) add logic to sphinx-needs to add the JSON builders to https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-smartquotes_excludes

[chrisjsewell]

Just some notes on thinking about how further optimise the code

• Don't run it:

  • Skipping steps:
    • ✅ JSON builder now skips all of the write phase
    • ✅ Using smartquotes = False skips the entire (slow) smartquote code
  • Caching steps:
    • ✅ The initialisation of the inline parser for sphinx_needs grid creation now cached
    • ✅ Jinja template creation for external_needs now cached

• Optimise it

  • refactor the logic
  • refactor into faster language (Rust / C)
    • The difficulty here is in the tradeoff between increased speed of code in the language, and decreased speed due to the translation layer between codes.
    • Also anything that is primarily regexes, probably won't be sped up

• Parallelise it