fix(incremental_to_absolute transform): fix LRU cache memory handling and emit cache metrics

[GreyLilac09]

Summary

• Create metrics for cache size, events, and evictions
• Always track cache size (for metrics tracking)
• Previously, the size for the event values for gauges (and other metrics) was always 0, which is inaccurate and messes up memory tracking
• Get the item after inserting to LRU cache, ensuring it moves to the back of the LRU cache and doesn't get evicted immediately if the cache is full
• Added some tests for incremental_to_absolute, absolute_to_incremental, and LRU cache capacity policy behavior

Vector configuration

data_dir: ./vector-data-dir
sources:
  s0:
    type: statsd
    address: "0.0.0.0:8127"
    mode: udp
  internal_metrics:
    scrape_interval_secs: 10
    type: internal_metrics
transforms:
  t0:
    type: incremental_to_absolute
    inputs:
      - s0
    cache:
      max_bytes: 5000
      max_events: 100
  filter_cache_metrics:
    type: filter
    inputs:
      - internal_metrics
    condition: '.name == "component_cache_evictions_total" || .name == "component_cache_bytes" || .name == "component_cache_events"'
sinks:
  blackhole:
    type: blackhole
    inputs:
      # - s0
      - t0
  file:
    type: file
    inputs:
      - filter_cache_metrics
    path: /vector-%Y-%m-%d.log
    encoding:
      codec: json

I then ran a statsd firehose to send arbitrary metrics to s0: https://github.com/stvp/statsd-firehose

How did you test this PR?

Memory usage w/ continual statsd pushes on a real deployment:

vector_component_allocated_bytes:

New metrics:

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

Notes

• Please read our Vector contributor resources.
• Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
• Some CI checks run only after we manually approve them.
  • We recommend adding a pre-push hook, please see this template.
  • Alternatively, we recommend running the following locally before pushing to the remote branch:
    • make fmt
    • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
    • make test
• After a review is requested, please avoid force pushes to help us review incrementally.
  • Feel free to push as many commits as you want. They will be squashed into one before merging.
  • For example, you can run git merge origin master and git push.
• If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
  run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

[GreyLilac09]

@pront @thomasqueirozb Hey what's the timeline on which you can have a look/get this approved?

[pront]

Here we have two better options:

1. Spawn a background task to do the reporting (requires synchronization and graceful shutdown)
2. or implement this pattern:

 loop {
                  tokio::select! {
                      _ = interval.tick() => {
						inner.emit_metrics();
                      },
					  maybe_event = task.next() => { // ...

[pront]

The allocated_bytes() method should return only heap-allocated memory, excluding stack memory.

[GreyLilac09]

What's your suggested fix here? If we set it to 0 as it is before the memory tracking won't work.

[pront]

We should probably use .size_of() in item_size() but I am going based on pub trait ByteSizeOf, didn't have time to dig into this PR.

[GreyLilac09]

@pront should be fixed now

[GreyLilac09]

@pront bump

[GreyLilac09]

@pront @thomasqueirozb Bump can you guys have a look please?