Question: why not use Cython or mypyc for all the code?

[alexconstsh]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

QUESTION

Describe the problem you're observing.

Hello. My assumption is that .pyx files are Cython files while .py are regular Python files.
I am curious as to why you would want to compile only specific parts with Cython?
Wouldn't it make sense to compile everything?

[ThomasWaldmann]

We use Cython for:

• interfacing to C libraries, external as well as self-made
• some performance critical code (if not using even C for that)

I never tried using Cython for all.

Using it for the not performance-critical parts of the code might have more disadvantages (compile time, binary size, debugging) than advantages though.

What might be more interesting is to identify more parts of the code that are perf critical.

[alexconstsh]

What might be more interesting is to identify more parts of the code that are perf critical.

What are the tools used with Python for this? As a C/C++ developer I am mainly familiar with callgrind tool from valgrind project.

[stuaxo]

It might be worth looking at mypyc if you are looking at something to process the rest of the code.

[ThomasWaldmann]

https://mypyc.readthedocs.io/en/latest/introduction.html

Good idea. But:

• we do not have that many type hints in the code yet
• mypy is not integrated into our workflow yet, see my still open mypy PR. add mypy #6862
• mypyc is alpha still, guess we want to wait a bit until it is at least beta.

[RonnyPfannschmidt]

When I experimented with mypyc on pluggy it yielded horrible results in terms of reliability, not something I'd Want in backups

[stuaxo]

Do you mean in terms of correctness or in terms of reliability for speed ups ?

I can imagine pluggy being more on the tricky end for something like mypyc so I'm sure they would welcome the bug reports.

[ThomasWaldmann]

Assessment by Claude Opus 4.6 (Thinking):

This is a resolved question/discussion from 2021–2022 with no actionable work items remaining.

The conclusions from the discussion still hold:

• Cython for everything: The cost (compile time, binary size, debugging difficulty) outweighs the benefit for non-performance-critical code. The current selective approach — using Cython for C library interfacing and performance-critical paths — is the right trade-off.
• mypyc: Still not mature or reliable enough for correctness-critical software like Borg. As noted by a contributor, it yielded poor reliability results in practice, which is unacceptable for backup software where data integrity is paramount.
• Profiling for more Cython candidates: This remains a valid idea in general, but is a standing engineering practice rather than a trackable issue.

Closing as not planned — the question has been thoroughly answered and there is nothing further to act on.