feat: create feature flag to disable default processors in Beat receivers

[andrzej-stencel]

What does this PR do?

Introduces a new feature flag default_processors with default value true. When set to false, the processors like add_host_metadata that are by default added to Beat receivers are disabled.

Why is it important?

We want to be able to disable the processors in Beat receivers so that they can be configured separately from Beat receivers. This will allow us to only use a single instance of e.g. add_kubernetes_metadata processor for multiple Beat receivers.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
  • Should this feature flag be configured in https://www.elastic.co/docs/reference/fleet/elastic-agent-standalone-feature-flags ?
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
  • Should this change be mentioned in the release notes?
• [ ] I have added an integration test or an E2E test

Disruptive User Impact

Unless the feature flag is explicitly configured, there is no change in behavior.

How to test this PR locally

• Prepare Agent configuration with agent::features::default_processors::enabled: false and with agent.internal.runtime.default: otel to use the OTel runtime.

agent:
  features:
    default_processors:
      enabled: false
  grpc:
    port: 6799 # Specify different port from 6789 to prevent conflict with the Agent installed on your machine.
  internal:
    runtime:
      default: otel

inputs:
  - id: agent-unique-input-id
    type: filestream
    paths:
      - /tmp/logs-1k.log

outputs:
  default:
    type: elasticsearch
    hosts:
      - localhost:9200
    username: elastic
    password: CHANGEME
    queue:
      mem:
        flush:
          timeout: 0

• Build Elastic Agent binary
• Run Agent with the above config
• Observe the logs in Elasticsearch. The logs should only have host name in the host field, no other details. This shows that the add_host_metadata processor that is included by default was disabled by the feature flag.

Related issues

• Relates Add a feature flag to disable injecting the default beat global processors when using the otel runtime #12267

[mergify]

This pull request does not have a backport label. Could you fix it @andrzej-stencel? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature-flag-default-processors upstream/feature-flag-default-processors
git merge upstream/main
git push upstream feature-flag-default-processors

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 9b642c6

Failed CI Steps

• Unit tests - macOS 15 ARM
• Unit tests - macOS 15 ARM
• Win2022:sudo:upgrade
• Win2022:non-sudo:default

History

• 💛 Build #33778 was flaky 864e78c
• 💔 Build #33758 failed fc123eb
• 💔 Build #33756 failed 4daf639
• 💔 Build #33577 failed 51fcbd9

cc @andrzej-stencel

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

The global processors aren't visible in the generated otel-merged.yaml since they are implicit. That makes it's hard to tell this works without inspecting the underlying data.

I would like to see a test that proves the global processors are in fact gone. If we could make them obvious in the translated config instead of implicit that would be nice and give us a way to test this without directly looking at the data or looking at logs we know come from these processors at startup.

[cmacknz]

The implicit default processors are only necessary when run as part of Elastic Agent, so for the beat receivers maybe we could start injecting them as part of config translation or allow controlling them as part of receiver creation.

For example the defaultProcessors function in Filebeat could instead be removed, and we inject the relevant processors section during translation: https://github.com/elastic/beats/blob/c9e93a708f90781a6817ce9d5f48f6576f59cc96/x-pack/filebeat/fbreceiver/factory.go#L33-L40

The only thing to be careful of is that the default processors need to stay in the same order in the pipeline (this affects whether processors after them can access fields they add, if they are last then this doesn't matter because nothing can access the fields and they need to stay last).

[cmacknz]

Testing manually, I can see the current implementation does work, where the full set of host metadata is not attached to events, you just can't confirm that with any automated testing from what I can see.

[andrzej-stencel]

The only thing to be careful of is that the default processors need to stay in the same order in the pipeline (this affects whether processors after them can access fields they add, if they are last then this doesn't matter because nothing can access the fields and they need to stay last).

If my understanding is correct, there is no way for the default global processors to be defined together with other global processors, rendering the problem of ordering nonexistent. See below for my train of thought and please let me know if I'm missing something:

In the current state there is no way to add global processors in Elastic Agent configuration. There is no "Processors" section in https://www.elastic.co/docs/reference/fleet/structure-config-file. Processors can be defined in the "Inputs" section, but those are input-level processors, not the global ones.

This means that when translating Agent config to OTel runtime config, there are no other processors that could be put either before or after the default processors - the default processors are the only global processors that can exist in the Beat receiver configuration.

If the user writes their Beat receiver config manually (bypassing Agent config translation to OTel runtime), the default processors are only added if there are no processors defined in the config, which also removes the possibility of the default processors clashing with user-defined processors.

@cmacknz please let me know if my understanding is correct or I'm missing something.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature-flag-default-processors upstream/feature-flag-default-processors
git merge upstream/main
git push upstream feature-flag-default-processors

[andrzej-stencel]

Let me move it to draft while working on #12493 and related changes.