Add prometheus metric reporting for ruby-kafka

[dhruvparanjape]

copied as is from upstream https://github.com/zendesk/ruby-kafka

[Copilot]

Pull Request Overview

This PR adds Prometheus metric reporting support for the ruby-kafka library by implementing a comprehensive monitoring system. The implementation includes metrics tracking for various Kafka operations including producer, consumer, and connection activities.

• Implements Prometheus metrics collection for Kafka operations using ActiveSupport subscribers
• Adds comprehensive test coverage for all monitored Kafka events and metrics
• Extends consumer batch processing to include last message create time for time lag calculations

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
lib/kafka/prometheus.rb	Core implementation of Prometheus metrics collection with subscriber classes for different Kafka components
spec/prometheus_spec.rb	Comprehensive test suite covering all Prometheus metrics and event scenarios
lib/kafka/consumer.rb	Minor enhancement to include last message create time in batch processing payload
Gemfile	Adds prometheus-client gem dependency for development/test environments

[nithyanandankm]

Overall looks good, nice one!

Is there an easy way to generate some metrics in staging to make sure that it works as expected?

[fcbr]

why are you only adding metrics to the consumer?

[dhruvparanjape]

Overall looks good, nice one!

Is there an easy way to generate some metrics in staging to make sure that it works as expected?

I would have to create a pre-release gem which isn't the easiest for this gem. Since this is an opt-in feature I am not sure if it would break anything. LMK if I am missing anything

[dhruvparanjape]

why are you only adding metrics to the consumer?

Um metrics are for both producers and consumers. The consumer was missing a data point so I added that.

[fcbr]

LGTM but I strongly recommend manually publishing a pre-release version and testing this before merging this.

You can just use gem release and use semver to mark it as -alpha and it should work fine. I've done this in the past - here's some reference docs. You need a PAT with the proper permissions.