Dependencies update for S3/HDFS datasources

[ostinru]

Hi, fellas!
I'd like to bump minor versions of dependencies for all S3 and HDFS/Hive datasources.

At this moment I would like to:

• bump minor versions - get low hanging fruits: fix CVEs (if any) and get bugfixes.
• add support for zstd compression support
• test in manually & in CI
• probably fix couple tests from pxf_fdw's pool of broken tests.

[MisterRaindrop]

Regarding CVE, I have been researching for a while.

PXF depends on Spring Boot. Currently, the Spring Boot 3.4.x series has no vulnerabilities. However, their minimum required JDK version is 17.

Hadoop currently likely only supports up to JDK 11.

PXF also depends on the JDK in Cloudberry. Currently, the kernel is JDK 8.

If we need to change the JDK, we must upgrade from Cloudberry to the surrounding Java components in the Cloudberry ecosystem. This change would be very extensive.

[ostinru]

Spring Boot has very short OSS support time (green boxes):

From vmware-pxf changelog it is clear that they are working with spring-boot-2.7.30.
And... from spring-boot changelog: it mostly dependencies updates.
And... almost all dependencies available (except enterprise spring-*).
So, we can live with spring-boot-2.7.x for a while with java 8..21.

However I would like to test it with java 17 (they said that every major JVM version gives several percent boost in performance) and with java 22+ (they said that JEP454 enables java to have faster compression/decompression libraries).

PXF also depends on the JDK in Cloudberry. Currently, the kernel is JDK 8.

f we need to change the JDK, we must upgrade from Cloudberry to the surrounding Java components in the Cloudberry ecosystem. This change would be very extensive.

Could you point on what in Cloudberry requires JVM?
Unless Cloudberry uses cloudberry-pxf java-libraries, it should be possible to run 2 different JVMs on the same machine.

Hadoop currently likely only supports up to JDK 11.

I will check hadoop client-libraries.

PS: java / spring upgrade is out of scope for this issue.

[tuhaihe]

Hi guys, could you see the details on this page: https://github.com/apache/cloudberry-pxf/security/dependabot? Maybe it can be a reference for security updates.

[MisterRaindrop]

Hi guys, could you see the details on this page: https://github.com/apache/cloudberry-pxf/security/dependabot? Maybe it can be a reference for security updates.

404 not found

[ostinru]

Hi guys, could you see the details on this page: https://github.com/apache/cloudberry-pxf/security/dependabot? Maybe it can be a reference for security updates.

We have this page in our fork. There are a lot of notifications for:

• automation tests - there is no urgency to fix
• spring/tomcat - there are vulnerabilities, however it is hard to exploit when PXF is running on the same host as cloudberry (it is the only supported option).
• issues in other components - it can be dangerous (e.g. security issue in avro) - that is what I am want to fix in this ticket.

[MisterRaindrop]

other cloudberry-contrib depends on jdk. see detail https://github.com/cloudberry-contrib/?q=&type=all&language=java&sort=

[MisterRaindrop]

Regarding the Spring issue, I suggest opening a new issue to discuss it ？