Race condition on provider installation with parallel_plan/apply enabled (Text file busy)

[uroja97]

Description

When enabling parallel_plan: true and parallel_apply: true in atlantis.yaml, we are experiencing concurrency issues with Terraform provider installation. Multiple parallel executions try to write/read to the same shared plugin cache directory simultaneously, resulting in text file busy errors or checksum mismatches.

It seems that even when using a shared plugin cache, concurrent terraform init or terraform plan operations conflict when accessing the provider binaries.

Steps to Reproduce

1. Enable parallel execution in atlantis.yaml:

   parallel_plan: true
   parallel_apply: true

2. Configure a shared plugin cache (e.g., via TF_PLUGIN_CACHE_DIR env var or .terraformrc).
3. Trigger a PR that runs multiple Terraform projects simultaneously (e.g., 5-10 projects) using the same providers.

Logs

│ Error: Failed to install provider
│ 
│ Error while installing hashicorp/azuread v3.7.0: open
│ /atlantis-data/plugin-cache/registry.terraform.io/hashicorp/azuread/3.7.0/linux_amd64/terraform-provider-azuread_v3.7.0_x5:
│ text file busy

And sometimes checksum errors:

│ Error: Required plugins are not installed
│ 
│ The installed provider plugins are not consistent with the packages
│ selected in the dependency lock file:
│   - registry.terraform.io/hashicorp/azurerm: the cached package for registry.terraform.io/hashicorp/azurerm 4.54.0 (in .terraform/providers) does not match any of the checksums recorded in the dependency lock file

Environment details

• Atlantis version: v0.37.1
• Terraform version: v1.13.5
• Atlantis server side config:
  • TF_PLUGIN_CACHE_DIR is set to a shared directory.

Workaround attempted

We had to implement a workaround in our atlantis.yaml to serialize the init phase and force a local download of providers (bypassing the cache) to avoid conflicts:

workflows:
  default:
    plan:
      steps:
        # Use flock to serialize init and disable cache to avoid symlink conflicts
        - run: flock /tmp/terraform_init.lock bash -c "rm -rf .terraform/providers && env -u TF_PLUGIN_CACHE_DIR TF_CLI_CONFIG_FILE=/dev/null terraform init -upgrade"
        - plan

Proposed Solution / Feature Request

It would be great if Atlantis could handle the locking mechanism for the provider cache internally when parallel mode is enabled, or provide a native way to serialize the init step while keeping plan/apply parallel.

[dosubot]

Hi @uroja97! I'm Dosu and I’m helping the atlantis team.

This is a known issue: Terraform's plugin cache (plugin_cache_dir) is not concurrency safe, so running parallel terraform init or plan with a shared cache can cause "text file busy" errors and checksum mismatches. Atlantis does not currently implement any internal locking or serialization for the init phase or provider cache access when parallel execution is enabled—this has been discussed in several issues and is a limitation of both Terraform and Atlantis as of v0.37.1 [#2412] [#2242] [#3547].

Workarounds include: disabling the plugin cache with the --use-tf-plugin-cache=false server flag (which avoids the race but increases provider download times) [docs], serializing terraform init with a lock (like your flock approach), using unique cache directories per project, or advanced OverlayFS-based solutions (which require privileged containers) [OverlayFS example]. Setting TF_PLUGIN_CACHE_MAY_BREAK_DEPENDENCY_LOCK_FILE=true can help in some cases, but doesn't fully resolve the race.

There is no built-in Atlantis feature to serialize only init while keeping plan/apply parallel; this has been requested but is not implemented. Disabling the plugin cache is the officially documented workaround for large parallel deployments, but it does impact performance [docs].

If you want more details on any workaround or implementation, let me know!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}