NIFI-15449 - NAR deletion blocks indefinitely when Python processor is initializing

[pvillard31]

Summary

NIFI-15449 - NAR deletion blocks indefinitely when Python processor is initializing

The issue is caused by:

• Deadlock between locks: a deadlock occurs between StandardPythonBridge and StandardExtensionDiscoveringManager:
  • Thread A (NAR deletion): Acquires StandardExtensionDiscoveringManager lock -> waits for StandardPythonBridge lock
  • Thread B (Processor initialization): Acquires StandardPythonBridge lock -> calls getNarDirectories() which waits for StandardExtensionDiscoveringManager lock
• Non-interruptible initialization: The Python process initialization (virtual environment creation, debugpy installation) runs in a tight loop with no mechanism to cancel or interrupt it when a shutdown is requested.

Changes:

• Break the deadlock: Pre-compute the NAR directories before acquiring the StandardPythonBridge synchronized block, ensuring locks are always acquired in a consistent order.
• Make initialization interruptible:
  • Add periodic isShutdown() checks during venv creation and dependency installation loops in PythonProcess
  • Add a CANCELLED state to AsyncLoadedProcessor.LoadState and a cancelLoading() method
  • Implement cancellation support in StandardPythonProcessorBridge that sets a flag checked during initialization
  • Track Python processes from the moment they're created (before start() is called) so they can be properly shut down during NAR deletion

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000
• Pull request contains commits signed with a registered key indicating Verified status

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using ./mvnw clean install -P contrib-check
  • JDK 21
  • JDK 25

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy
• New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• Documentation formatting appears as expected in rendered files