Deadlock issue when using clj-http from JVM/Jetty virtual threads limited by number of CPU cores

[claj]

We had a problem using clj-http in the context of Jetty using virtual threads (JDK version 25). On a docker container with only 2 cpus (via docker resource limit) an org.apache.http.conn.ConnectionPoolTimeoutException, "Timeout waiting for connection from pool" was thrown after approx 20 seconds.

According to a surprisingly confident Claude LLM the root cause is:

Apache HttpClient 4.x uses synchronized blocks in its connection pool (AbstractNIOConnPool/connection management). With virtual threads, this causes thread pinning — virtual threads get pinned to carrier OS threads while holding or waiting for locks, and with only 2 CPU cores (= 2 ForkJoinPool workers), you can deadlock: all carrier threads are occupied by pinned VTs waiting on a lock, and the VT that holds the lock can't be scheduled.

According to this war story the explanation sounds plausible: Java Virtual Threads — pitfalls to look out for!

Possible fix: future JVM improvements

According to the article future JVM versions might will be able to treat the synchronized keyword correctly even in virtual threads, which would solve the problem.

(Non?-)Possible fix: update Apache Http client libs

I see there has been efforts attempting to update the Apache Http client libraries, but with a lot of potentially breaking functionality and large changes of the APIs. Since the JVM has its built in HTTP client (and the wrapper Hato for Clojure) I guess the Apache HTTP client also might be turning its focus on narrow niche use cases with more knobs exposed.

Our Workaround

In our quite standard use case we had the possibility to change the http client to Hato which seems to work better with virtual threads.