Concurrent map iteration and write in `mergeNonProtocolHeaders`

[elefantepequeno]

Describe the bug

Concurrent map iteration and write in mergeNonProtocolHeaders.

To Reproduce

This was randomly caught by my local tests, don't think I can reproduce this.
Should be easy to trace manually though.

Environment (please complete the following information):

• connect-go version or commit: v1.18.1
• go version: go version go1.25.4 linux/arm64
• your complete go.mod:

go 1.25.1

require (
	connectrpc.com/connect v1.18.1
	connectrpc.com/cors v0.1.0
	connectrpc.com/otelconnect v0.7.2
	github.com/VideoCentrix/proto v0.0.0
	github.com/aws/aws-sdk-go-v2/config v1.30.2
	github.com/aws/aws-sdk-go-v2/credentials v1.18.2
	github.com/aws/aws-sdk-go-v2/service/s3 v1.85.1
	github.com/aws/smithy-go v1.22.5
	github.com/caarlos0/env/v11 v11.3.1
	github.com/deckarep/golang-set/v2 v2.8.0
	github.com/exaring/otelpgx v0.9.3
	github.com/google/uuid v1.6.0
	github.com/hashicorp/golang-lru/v2 v2.0.7
	github.com/jackc/pgx-zap v0.0.0-20221202020421-94b1cb2f889f
	github.com/jackc/pgx/v5 v5.7.5
	github.com/pgvector/pgvector-go v0.3.0
	github.com/roboslone/go-framework v1.2.2
	github.com/roboslone/go-ui-fs v1.0.2
	github.com/rs/cors v1.11.1
	github.com/stretchr/testify v1.11.1
	github.com/twpayne/go-geom v1.6.1
	github.com/twpayne/pgx-geom v0.0.2
	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.62.0
	go.opentelemetry.io/otel v1.37.0
	go.opentelemetry.io/otel/sdk v1.37.0
	go.opentelemetry.io/otel/sdk/metric v1.37.0
	go.opentelemetry.io/otel/trace v1.37.0
	go.uber.org/zap v1.27.0
	golang.org/x/net v0.41.0
	google.golang.org/protobuf v1.36.6
)

[iamrajiv]

hi @elefantepequeno can i work on this?

[elefantepequeno]

@iamrajiv Sure, I was only reporting.

[jhump]

@elefantepequeno, do you happen to remember whether it was the iteration over from or the write to to inside mergeNonProtocolHeaders that was the problem? I don't suppose you still have a stack-trace handy? That would likely be helpful, to see which of the calls to mergeNonProtocolHeaders was the problem.

I've done a very quick scan of the call sites, and none of them are obviously problematic. I think the most likely case I can think of is something with a streaming RPC, where the server handler might spin up a goroutine that calls stream.Send (which will flush response headers to the client if necessary, and call mergeNonProtocolHeaders), which could happen concurrently with something on the main handler goroutine iterating over the same map.

A code example would be helpful since I suspect it could be specific to a formulation of the handler code and its use of goroutines.

[elefantepequeno]

I never had a stacktrace, I don't think golang generates it for concurrent map access.

[jhump]

@elefantepequeno, oh, I figured maybe this was a race detector issue (which will print a goroutine stack dump). How did you detect/conclude this was happening?

[elefantepequeno]

I just ran go test, without race detector. At the moment I'm not able to run any tests whatsoever, at least for a week. Running connectrpc tests with race detector might yield something useful though.