[SPARK-56276] Fix file descriptor leak in SparkAppResourceSpecFactory#597
[SPARK-56276] Fix file descriptor leak in SparkAppResourceSpecFactory#597j1wonpark wants to merge 2 commits intoapache:mainfrom
Conversation
Use try-with-resources for OutputStreamWriter in createLocalFileForPodTemplateSpec to ensure streams are closed even when an exception is thrown during write. Also narrow the catch clause in deleteLocalFileFromPathKey from Throwable to SecurityException, which is the only exception File.delete() can throw. Add a test to verify that temp files written for pod template specs are cleaned up after buildResourceSpec completes. Signed-off-by: Jiwon Park <jpark92@outlook.kr>
dongjoon-hyun
left a comment
There was a problem hiding this comment.
The test case should fail without your patch, @j1wonpark . It's the fundamental principle of test-driven development.
Currently, the main body looks like a simple try-with-resource improvement, but the corresponding test case seems insufficient to provide any proof or a test coverage to protect your contribution from the future regression.
|
Thank you for the feedback. I tried writing a test that fails without the patch, but since Would it be acceptable to proceed without an additional test for this, or do you have suggestions on how to approach this? |
|
In the community, we remove the misleading test case in that case, @j1wonpark . For example, we handle independent themes as two PRs.
|
…ctory Remove testBuildResourceSpecWritesAndCleansTempFilesForPodTemplateSpec as it passes with or without the try-with-resources fix, providing no regression protection for the file descriptor leak patch. Signed-off-by: Jiwon Park <jpark92@outlook.kr>
|
Thank you for the guidance, @dongjoon-hyun. I removed the misleading test case ( This PR now contains only the |
What changes were proposed in this pull request?
OutputStreamWriterincreateLocalFileForPodTemplateSpectoguarantee streams are closed even when an exception is thrown during the write operation.
"UTF-8"string literal withStandardCharsets.UTF_8for type safety.catch (Throwable t)clause indeleteLocalFileFromPathKeytocatch (SecurityException e), which is the only exceptionFile.delete()can throw.Why are the changes needed?
FileOutputStreamandOutputStreamWriterwere opened without try-with-resources.If
writer.write()orModelUtils.asJsonString()threw an exception, the streams wouldremain open, causing file descriptor leaks. Over time this could exhaust the OS file
descriptor limit in long-running operator deployments.
Catching
ThrowableindeleteLocalFileFromPathKeyis an anti-pattern that swallowsJVM-level errors such as
OutOfMemoryErrorandVirtualMachineError.Does this PR introduce any user-facing change?
No.
How was this patch tested?
testBuildResourceSpecWritesAndCleansTempFilesForPodTemplateSpectoSparkAppResourceSpecFactoryTestto verify that temp files created for driver and executorpod template specs are deleted after
buildResourceSpeccompletes../gradlew :spark-operator:test,checkstyleMain,checkstyleTest,pmdMain,spotbugsMain, andspotlessCheck— all passed.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.6