Skip to content

[SPARK-56276] Fix file descriptor leak in SparkAppResourceSpecFactory#597

Open
j1wonpark wants to merge 2 commits intoapache:mainfrom
j1wonpark:fix/stream-resource-leak
Open

[SPARK-56276] Fix file descriptor leak in SparkAppResourceSpecFactory#597
j1wonpark wants to merge 2 commits intoapache:mainfrom
j1wonpark:fix/stream-resource-leak

Conversation

@j1wonpark
Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

  • Use try-with-resources for OutputStreamWriter in createLocalFileForPodTemplateSpec to
    guarantee streams are closed even when an exception is thrown during the write operation.
  • Replace the "UTF-8" string literal with StandardCharsets.UTF_8 for type safety.
  • Narrow the catch (Throwable t) clause in deleteLocalFileFromPathKey to
    catch (SecurityException e), which is the only exception File.delete() can throw.

Why are the changes needed?

FileOutputStream and OutputStreamWriter were opened without try-with-resources.
If writer.write() or ModelUtils.asJsonString() threw an exception, the streams would
remain open, causing file descriptor leaks. Over time this could exhaust the OS file
descriptor limit in long-running operator deployments.

Catching Throwable in deleteLocalFileFromPathKey is an anti-pattern that swallows
JVM-level errors such as OutOfMemoryError and VirtualMachineError.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

  • Added testBuildResourceSpecWritesAndCleansTempFilesForPodTemplateSpec to
    SparkAppResourceSpecFactoryTest to verify that temp files created for driver and executor
    pod template specs are deleted after buildResourceSpec completes.
  • Ran ./gradlew :spark-operator:test, checkstyleMain, checkstyleTest, pmdMain,
    spotbugsMain, and spotlessCheck — all passed.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Sonnet 4.6

Use try-with-resources for OutputStreamWriter in createLocalFileForPodTemplateSpec
to ensure streams are closed even when an exception is thrown during write.
Also narrow the catch clause in deleteLocalFileFromPathKey from Throwable
to SecurityException, which is the only exception File.delete() can throw.

Add a test to verify that temp files written for pod template specs are
cleaned up after buildResourceSpec completes.

Signed-off-by: Jiwon Park <jpark92@outlook.kr>
Copy link
Copy Markdown
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test case should fail without your patch, @j1wonpark . It's the fundamental principle of test-driven development.

Currently, the main body looks like a simple try-with-resource improvement, but the corresponding test case seems insufficient to provide any proof or a test coverage to protect your contribution from the future regression.

@j1wonpark
Copy link
Copy Markdown
Contributor Author

Thank you for the feedback. I tried writing a test that fails without the patch, but since OutputStreamWriter is created inside the private method, there's no way to inject a spy to verify close() is called on the exception path. Restructuring the production code solely for testability felt like over-engineering for this change.

Would it be acceptable to proceed without an additional test for this, or do you have suggestions on how to approach this?

@dongjoon-hyun
Copy link
Copy Markdown
Member

In the community, we remove the misleading test case in that case, @j1wonpark .

For example, we handle independent themes as two PRs.

  • We can accept your try-with-resource improvement as a better practice without a test case since it's a trivial Java recommendation.
  • We can accept your happy scenario test case as a new test improvement independently (if it's needed).

…ctory

Remove testBuildResourceSpecWritesAndCleansTempFilesForPodTemplateSpec
as it passes with or without the try-with-resources fix, providing no
regression protection for the file descriptor leak patch.

Signed-off-by: Jiwon Park <jpark92@outlook.kr>
@j1wonpark
Copy link
Copy Markdown
Contributor Author

Thank you for the guidance, @dongjoon-hyun.

I removed the misleading test case (testBuildResourceSpecWritesAndCleansTempFilesForPodTemplateSpec) since it passes with or without the patch and does not protect against regression.

This PR now contains only the try-with-resources improvement and related cleanup (StandardCharsets.UTF_8, narrowed catch clause).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants