-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Open
Labels
Description
What is the issue?
We are seeing 2 issues:
- When using gRPC, many
rpc error: code = ResourceExhausted desc = stream terminated by RST_STREAM with error code: ENHANCE_YOUR_CALMerrors, this appears to be fixed by fix: adjust expiring reset stream limits hyperium/h2#858 (released in v0.4.12) - When using regular HTTP2, streams entering a deadlock state over a connection, which also appears to have a fix fix: do not assign capacity for pending streams hyperium/h2#860 (released in v0.4.13)
How can it be reproduced?
- The reset issue is described in Downstream reports of increased go aways with "too_many_internal_resets" hyperium/h2#856
- The deadlock issue is described in this discussion http2: proxy does not open new connections when max concurrent streams is reached #14800
Logs, error output, etc
- Reset issue
WARN ThreadId(01) h2::proto::streams::streams: locally-reset streams reached limit (1024) INFO ThreadId(01) linkerd_app_core::errors::respond: gRPC request failed error=http2 error error.sources=[connection error detected: detected excessive load generating behavior (b"too_many_internal_resets")] INFO ThreadId(01) linkerd_app_core::errors::respond: gRPC request failed error=client 192.168.1.3:50048: server: 192.168.1.12:8080: server 192.168.1.12:8080: service ourgrpcservice.ournamespace:80: http2 error error.sources=[server 192.168.1.12:8080: service ourgrpcservice.ournamespace:80: http2 error, http2 error, connection error detected: detected excessive load generating behavior (b"too_many_internal_resets")] - Deadlock issue, please see http2: proxy does not open new connections when max concurrent streams is reached #14800
output of linkerd check -o short
/
Environment
- Kubernetes Version: v1.33.5
- Cluster Environment: GKE
- Host OS: Google Container-Optimized OS
- Linkerd Version: edge-25.9.4
Possible solution
The proxy is currently built with h2 v0.4.11:
Could the h2 crate be upgraded to v0.4.13+, please?
Additional context
No response
Would you like to work on fixing this bug?
maybe
Reactions are currently unavailable