Memory Leak in Protobuf Decode Processor

[josephpd3]

Details

• Our team recently migrated to redpanda-data/connect (github.com/redpanda-data/benthos/v4 v4.63.0 v4.72.0) from previous benthos (v4.27-x ) for a lot of our workload
• We utilize protobuf decode on incoming data

- protobuf:
	operator: "to_json"
	message: "PushDataV3ApiWrapper"
	discard_unknown: false
	use_proto_names: false
	import_paths:
	  - /etc/benthos/exchange_methods/PushDataV3ApiWrapper.proto

• Noted, particularly on high throughput jobs (~4-5k RPS) that processes running our pipelines would continue to eat memory until they either were OOMKilled or led to larger issues with memory-pressure on host nodes
  • Same jobs would hover around ~100MB memory in steady state

- Before and after shown above, before reverting image to Benthos v4.27

[rockwotj]

What version of Redpanda Connect did you try?

[rockwotj]

In v4.69.0, we introduced a new protobuf parser, I wonder if the issue lies there. If v4.68.0 didn't have a leak then that would prove it.

[josephpd3]

@rockwotj we are on v4.72.0 currently