feat(otel/translate): explicitly configure default processors for Beat receivers

[andrzej-stencel]

What does this PR do?

The default processors are now added to the Beat receivers' config explicitly during config translation.

Why is it important?

Previously, the default processors were added to Beat receivers implicitly, when there were no processors configured in the Beat receiver's configuration. This was unnecessary and potentially confusing, making the rendered configuration not matching the actual behavior. Also see #12334 (comment)

Explicitly mentioning the default processors in the config makes it easier to reason about the effective configuration of the Beat receivers.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

How to test this PR locally

• Prepare Agent configuration with agent.internal.runtime.default: otel to use the OTel runtime.

agent:
  grpc:
    port: 6799 # Specify different port from 6789 to prevent conflict with the Agent installed on your machine.
  internal:
    runtime:
      default: otel

inputs:
  - id: agent-unique-input-id
    type: filestream
    paths:
      - /tmp/logs-1k.log

outputs:
  default:
    type: elasticsearch
    hosts:
      - localhost:9200
    username: elastic
    password: CHANGEME
    queue:
      mem:
        flush:
          timeout: 0

• Build Elastic Agent binary
• Run Agent with the above config
• Generate diagnostics and inspect the edot/otel-merged-actual.yaml file from the diagnostics package. The Beat receiver configs like receivers::filebeatreceiver/_agent-component/filestream-default should have the processors node specified in their config with the default processors in it.
• Observe the logs in Elasticsearch to confirm that the default processors are still doing their job. The logs should have multiple fields under host field (for example host.architecture). This shows that the add_host_metadata processor is still included.

Related issues

• Relates Add a feature flag to disable injecting the default beat global processors when using the otel runtime #12267

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[leehinman]

@andrzej-stencel to really test this don't we also have to make a change in beats so the default processors aren't included?

https://github.com/elastic/beats/blob/d282e8bd0d73036f2f6bcf4dfb38ce6283a257b8/x-pack/filebeat/fbreceiver/factory.go#L40

[andrzej-stencel]

@andrzej-stencel to really test this don't we also have to make a change in beats so the default processors aren't included?

https://github.com/elastic/beats/blob/d282e8bd0d73036f2f6bcf4dfb38ce6283a257b8/x-pack/filebeat/fbreceiver/factory.go#L40

Yes I agree we should remove that code, but I don't consider it a blocker to this PR. This PR makes it so the default processors are specified explicitly, which makes the code in Beats no-op - the processors are not injected there, because they are already explicitly defined.

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 1c4c29b

Failed CI Steps

• Unit tests - macOS 15 ARM
• Unit tests - macOS 15 ARM
• x86_64:sudo: default - platform-ingest-elastic-agent-debian-11-1767879728
• x86_64:sudo: upgrade - platform-ingest-elastic-agent-debian-11-1767879728

History

cc @andrzej-stencel

[leehinman]

@andrzej-stencel to really test this don't we also have to make a change in beats so the default processors aren't included?
https://github.com/elastic/beats/blob/d282e8bd0d73036f2f6bcf4dfb38ce6283a257b8/x-pack/filebeat/fbreceiver/factory.go#L40

Yes I agree we should remove that code, but I don't consider it a blocker to this PR. This PR makes it so the default processors are specified explicitly, which makes the code in Beats no-op - the processors are not injected there, because they are already explicitly defined.

I was just thinking for as a step for the manual testing you have in this PR, not a proposed code change for this PR. The code in the beat receiver will always inject the global processors, so I'm not sure you would ever get documents that didn't have the fields from add_host_metadata processor.

[cmacknz]

Yes, this PR looks correct but you won't know for sure until the change in Beat is made to stop configuring these twice.

[cmacknz]

We could merge this PR, then remove the processors from Beats, which would come in via a separate beats bump PR. If we know it's going to work via manual testing this is pretty low risk, the document equivalence tests would find that host was missing for example.

The other option is to do the beats bump PR and update beats in this PR, only consequence of that is the automated beat bump PRs will fail until this merges. This is probably more disruptive.

[andrzej-stencel]

Let me merge this and prepare the Beats PR removing injecting processors there.