Ansible Playbook Performance Profiling and Optimization

[parithosh]

Background

During some OOO digging, discovered that Ansible's built-in profiling callbacks provide valuable insights into playbook performance bottlenecks.

Current Findings

Using ANSIBLE_CALLBACKS_ENABLED="profile_tasks,profile_roles" on a single bootnode reveals both role level and task level time breakdowns

e.g:
Role Execution Times:

• lighthouse: 14.00s

Task-Level Breakdown:

• ethpandaops.general.lighthouse : Run lighthouse container - 10.71s
• ethpandaops.general.ethereum_node_fact_discovery : Get consensus node identity - 9.84s

Proposal

1. Basic Profiling Campaign

   • Run profiling against 50+ nodes to establish statistical baseline (likely one of the latest devnets during first setup time)
   • Identify which tasks are consistently slow vs. variable
   • Distinguish between unfixable tasks (e.g., docker run commands) and optimizable ones

2. Enhanced Profiling with ansible-runner

   ANSIBLE_CONFIG=./ansible.cfg ansible-runner run . -p playbook.yaml \
     --cmdline "--tags ethereum --limit bootnode" --artifact-dir ./artifacts
   This generates detailed artifacts with start/stop durations per task. We need a performance visualization tool that uses the files that are generated to showcase exactly the worst case tasks and roles. 
   

One or both approaches should give us enough info to figure out where we need to spend time optimising our ansible stack for 1k+ devnets.

[parithosh]

This should also allow us to benchmark which strategy (free) or if mitogen is actually worth spending time investigating. We need to be able to do ~1k+ devnets much faster than before in order to do it more often, so this might be worth spending some dev effort on.

[parithosh]

For the task: ANSIBLE_CALLBACKS_ENABLED="profile_tasks,profile_roles" ansible-playbook playbook.yaml -t ethereum_node_fact_discovery

We see that ethpandaops.general.ethereum_node_fact_discovery : Turn EL nodekey into enode takes ~24s out of a total playbook runtime of 38s. This could likely be due to missing batching or some missing optimisation.

[parithosh]

Two changes that i'd propose:
modify ansible.cfg to include this line: vars_plugin_stage = inventory

modify your linux setup (or whatever the equivalent mac one is) to increase file descriptors:

# Permanent - add to /etc/security/limits.conf on a linux machine
soft nofile 65536
hard nofile 65536

By default in linux, the descriptor limit is 1024. Ansible uses a few descriptors per fork and just sort of hangs when there are none free.

[parithosh]

proposed mitogen related changes:
to ansible.cfg

strategy             = mitogen_linear
strategy_plugins     = vendor/mitogen-0.3.27/ansible_mitogen/plugins/strategy

[mitogen]
# Optional: Adjust the size of the connection pool (default is 16)
# Increase if you have many hosts
# mitogen_pool_size = 50

switch ethereum_node related roles to strategy: mitogen_free

Both of these have shown an insane speedup and successful complation of the tasks.