Skip to content

Conversation

@macatizm
Copy link

Refactors docker-compose configuration to split environment variables into separate, organized files.

Changes:

  • Split env vars into categorized files (global, server, security, database, redis, storage, vectorstore)
  • Replace large YAML anchor with env_file references
  • Update generation script to create env files
  • Makes docker-compose.yaml cleaner and easier to maintain

Fixes #31572

@gemini-code-assist
Copy link
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. refactor labels Jan 27, 2026
- Split environment variables into categorized files (global, server, security, database, redis, storage, vectorstore)
- Replace large YAML anchor with env_file references using x-shared-config
- Update generate_docker_compose script to create env files from examples
- Makes docker-compose.yaml cleaner and more maintainable

Fixes langgenius#31572
@macatizm macatizm force-pushed the refactor/docker-compose-env-split branch from d79644e to 0882f1e Compare January 27, 2026 05:10
- Update test to read from env files instead of x-shared-env anchor
- Matches the new docker-compose structure using env_file
- Reorganize into: core-services, security, databases, vectorstores, infrastructure
- Core Services: api, worker, worker_beat, web, sandbox, plugin_daemon
- Databases: db_postgres, db_mysql, redis
- Vector Stores: all vector store services
- Infrastructure: nginx, certbot, ssrf_proxy, etc.
- Security: general security configurations

Updates langgenius#31572
@macatizm macatizm force-pushed the refactor/docker-compose-env-split branch from 0468a18 to c28eb0f Compare January 27, 2026 06:32
- Split into per-module structure: core-services/api, core-services/worker, etc.
- Core services: shared.env (for api/worker/worker-beat), api.env, worker.env, worker-beat.env, web.env, sandbox.env, plugin-daemon.env
- Databases: db-postgres.env, db-mysql.env, redis.env
- Vector stores: separate file per vector store service
- Infrastructure: separate file per infrastructure service
- Update docker-compose-template.yaml to reference per-module files
- Update generate_docker_compose and test to handle recursive structure

Updates langgenius#31572
- Remove core-services.env, databases.env, infrastructure.env, vectorstores.env
- Keep security.env as it's still referenced in docker-compose
- All variables are now in per-module structure
- Add VECTOR_STORE, WEAVIATE_*, QDRANT_*, OCEANBASE_*, PGVECTOR_*, MILVUS_* variables
- These are used by api/worker services and should be in shared.env
- Fixes pytest config test failure
Copy link
Member

@crazywoola crazywoola left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI. These *.env should not be committed into the main branch. Only *.example should be added.

In order to keep the start command clean. I think we need a way to simplify the command as well.

cd dify
cd docker
cp .env.example .env <= This line
docker compose up -d

- Remove all .env files from git (keep only .env.example files)
- Add .gitignore to ignore .env files
- Add setup_env.sh script to copy all .env.example files to .env
- Simplifies setup: users run './setup_env.sh' instead of copying many files manually
- Replace manual 'cp .env.example .env' with './setup_env.sh'
- Simplifies setup for users with per-module env file structure
- generate_docker_compose already creates .env files from .env.example
- This ensures .env files exist in CI/CD environments
- Fixes docker-compose error about missing env files
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Refactors the Docker Compose environment configuration by splitting the previous large shared YAML environment anchor into multiple categorized envs/**/*.env(.example) files and updating generation/setup workflows accordingly (Fixes #31572).

Changes:

  • Introduces categorized docker/envs/**.env.example files and updates Compose to load them via env_file.
  • Updates docker/generate_docker_compose and adds docker/setup_env.sh to generate/copy .env files from examples.
  • Adjusts config parity test to read the new split env files instead of the removed shared YAML anchor.

Reviewed changes

Copilot reviewed 41 out of 41 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
docker/setup_env.sh Adds setup script to copy .env.example.env (and envs examples → envs envs).
docker/generate_docker_compose Stops injecting shared YAML env anchor; adds env-file generation from examples.
docker/envs/security.env.example New split security env example file.
docker/envs/core-services/api.env.example New API-specific env example.
docker/envs/core-services/plugin-daemon.env.example New plugin-daemon env example.
docker/envs/core-services/sandbox.env.example New sandbox env example.
docker/envs/core-services/shared.env.example New shared API/worker env example (large split from old anchor).
docker/envs/core-services/web.env.example New web env example.
docker/envs/core-services/worker-beat.env.example New worker-beat env example.
docker/envs/core-services/worker.env.example New worker env example.
docker/envs/databases/db-mysql.env.example New MySQL env example.
docker/envs/databases/db-postgres.env.example New Postgres env example.
docker/envs/databases/redis.env.example New Redis env example.
docker/envs/infrastructure/certbot.env.example New certbot env example.
docker/envs/infrastructure/etcd.env.example New etcd env example.
docker/envs/infrastructure/milvus-standalone.env.example New milvus-standalone env example.
docker/envs/infrastructure/minio.env.example New minio env example.
docker/envs/infrastructure/nginx.env.example New nginx env example.
docker/envs/infrastructure/ssrf-proxy.env.example New ssrf-proxy env example.
docker/envs/vectorstores/chroma.env.example New Chroma env example.
docker/envs/vectorstores/couchbase.env.example New Couchbase env example.
docker/envs/vectorstores/elasticsearch.env.example New Elasticsearch env example.
docker/envs/vectorstores/iris.env.example New Iris env example.
docker/envs/vectorstores/matrixone.env.example New MatrixOne env example.
docker/envs/vectorstores/milvus.env.example New Milvus env example.
docker/envs/vectorstores/myscale.env.example New MyScale env example.
docker/envs/vectorstores/oceanbase.env.example New OceanBase env example.
docker/envs/vectorstores/opengauss.env.example New OpenGauss env example.
docker/envs/vectorstores/opensearch.env.example New OpenSearch env example.
docker/envs/vectorstores/oracle.env.example New Oracle env example.
docker/envs/vectorstores/pgvecto-rs.env.example New pgvecto-rs env example.
docker/envs/vectorstores/pgvector.env.example New pgvector env example.
docker/envs/vectorstores/qdrant.env.example New Qdrant env example.
docker/envs/vectorstores/seekdb.env.example New SeekDB env example.
docker/envs/vectorstores/vastbase.env.example New Vastbase env example.
docker/envs/vectorstores/weaviate.env.example New Weaviate env example.
docker/docker-compose.yaml Switches to shared anchor(s) that use env_file instead of huge YAML env anchor.
docker/docker-compose-template.yaml Updates template to match new env_file-based configuration.
docker/.gitignore Ignores generated .env files while keeping .env.example tracked.
dev/pytest/pytest_config_tests.py Updates docker-compose config-key discovery to read split env example files.
README.md Updates Docker Compose setup instructions to use ./setup_env.sh.
Comments suppressed due to low confidence (7)

docker/docker-compose-template.yaml:247

  • env_file is added, but core settings are also set under environment using ${...} (e.g., CONSOLE_API_URL, APP_API_URL). Since env_file doesn’t feed Compose interpolation and environment overrides env_file, changes to envs/core-services/web.env won’t affect these values. To make the split env files effective, remove the duplicated environment entries or define the exact container env vars in the env file and rely on env_file only.
    env_file:
      - ./envs/core-services/web.env
      - ./envs/security.env
    environment:
      CONSOLE_API_URL: ${CONSOLE_API_URL:-}
      APP_API_URL: ${APP_API_URL:-}
      AMPLITUDE_API_KEY: ${AMPLITUDE_API_KEY:-}
      NEXT_PUBLIC_COOKIE_DOMAIN: ${NEXT_PUBLIC_COOKIE_DOMAIN:-}
      SENTRY_DSN: ${WEB_SENTRY_DSN:-}

docker/docker-compose-template.yaml:144

  • The template introduces env_file for shared config, but this service still sets many values via ${...} interpolation under environment (e.g., PLUGIN_DAEMON_TIMEOUT, INNER_API_KEY_FOR_PLUGIN). Compose interpolation reads from the project .env/shell and does not use service env_file, and any keys repeated under environment override env_file. As a result, editing envs/**.env won’t actually change these settings. Prefer defining the final container env var names directly in the env files and removing the duplicated environment entries, or keep these vars in .env and don’t include them in env_file.
    <<: *shared-api-worker-config
    image: langgenius/dify-api:1.11.4
    environment:
      # Startup mode, 'api' starts the API server.
      MODE: api
      SENTRY_DSN: ${API_SENTRY_DSN:-}
      SENTRY_TRACES_SAMPLE_RATE: ${API_SENTRY_TRACES_SAMPLE_RATE:-1.0}
      SENTRY_PROFILES_SAMPLE_RATE: ${API_SENTRY_PROFILES_SAMPLE_RATE:-1.0}
      PLUGIN_REMOTE_INSTALL_HOST: ${EXPOSE_PLUGIN_DEBUGGING_HOST:-localhost}
      PLUGIN_REMOTE_INSTALL_PORT: ${EXPOSE_PLUGIN_DEBUGGING_PORT:-5003}
      PLUGIN_MAX_PACKAGE_SIZE: ${PLUGIN_MAX_PACKAGE_SIZE:-52428800}
      PLUGIN_DAEMON_TIMEOUT: ${PLUGIN_DAEMON_TIMEOUT:-600.0}
      INNER_API_KEY_FOR_PLUGIN: ${PLUGIN_DIFY_INNER_API_KEY:-QaHbTe77CtuXmsfyhR7+vRjI/+XbV1AaFy691iy+kGDv2Jvy0/eAh8Y1}

docker/docker-compose-template.yaml:369

  • Same problem here: env_file is added but the container vars are set via ${...} interpolation (e.g., API_KEY: ${SANDBOX_API_KEY:-...}), which won’t pick up values from envs/core-services/sandbox.env. Consider putting API_KEY, GIN_MODE, etc. directly into the sandbox env file and dropping the environment mappings, or document that these must come from the project .env instead.
    env_file:
      - ./envs/core-services/sandbox.env
      - ./envs/security.env
    environment:
      # The DifySandbox configurations
      # Make sure you are changing this key for your deployment with a strong key.
      # You can generate a strong key using `openssl rand -base64 42`.
      API_KEY: ${SANDBOX_API_KEY:-dify-sandbox}
      GIN_MODE: ${SANDBOX_GIN_MODE:-release}
      WORKER_TIMEOUT: ${SANDBOX_WORKER_TIMEOUT:-15}
      ENABLE_NETWORK: ${SANDBOX_ENABLE_NETWORK:-true}
      HTTP_PROXY: ${SANDBOX_HTTP_PROXY:-http://ssrf_proxy:3128}
      HTTPS_PROXY: ${SANDBOX_HTTPS_PROXY:-http://ssrf_proxy:3128}
      SANDBOX_PORT: ${SANDBOX_PORT:-8194}

docker/docker-compose-template.yaml:400

  • env_file is added for plugin_daemon, but critical values are still sourced via ${...} interpolation (e.g., SERVER_KEY, DIFY_INNER_API_URL). Since env_file doesn’t influence interpolation and environment overrides env_file, updating envs/core-services/plugin-daemon.env won’t affect these settings. Either rely on env_file by defining SERVER_KEY/DIFY_INNER_API_URL directly there, or remove the env file entries and keep using .env interpolation consistently.
    env_file:
      - ./envs/core-services/shared.env
      - ./envs/core-services/plugin-daemon.env
      - ./envs/security.env
      - ./envs/databases/db-postgres.env
      - ./envs/databases/db-mysql.env
      - ./envs/databases/redis.env
    networks:
      - ssrf_proxy_network
      - default
    environment:
      DB_DATABASE: ${DB_PLUGIN_DATABASE:-dify_plugin}
      SERVER_PORT: ${PLUGIN_DAEMON_PORT:-5002}
      SERVER_KEY: ${PLUGIN_DAEMON_KEY:-lYkiYYT6owG+71oLerGzA7GXCgOT++6ovaezWAjpCjf+Sjc3ZtU+qUEi}
      MAX_PLUGIN_PACKAGE_SIZE: ${PLUGIN_MAX_PACKAGE_SIZE:-52428800}
      PPROF_ENABLED: ${PLUGIN_PPROF_ENABLED:-false}
      DIFY_INNER_API_URL: ${PLUGIN_DIFY_INNER_API_URL:-http://api:5001}
      DIFY_INNER_API_KEY: ${PLUGIN_DIFY_INNER_API_KEY:-QaHbTe77CtuXmsfyhR7+vRjI/+XbV1AaFy691iy+kGDv2Jvy0/eAh8Y1}

docker/docker-compose.yaml:252

  • env_file is added, but the same values are also set in environment using ${...} interpolation (e.g., CONSOLE_API_URL, APP_API_URL). Compose interpolation reads from the project .env/shell, not from service env_file, and environment overrides env_file, so changes to envs/core-services/web.env won’t affect these variables. To make the split env files effective, remove these duplicated environment entries (or change the env file to define the exact container env var names and rely on env_file only).
    env_file:
      - ./envs/core-services/web.env
      - ./envs/security.env
    environment:
      CONSOLE_API_URL: ${CONSOLE_API_URL:-}
      APP_API_URL: ${APP_API_URL:-}
      AMPLITUDE_API_KEY: ${AMPLITUDE_API_KEY:-}
      NEXT_PUBLIC_COOKIE_DOMAIN: ${NEXT_PUBLIC_COOKIE_DOMAIN:-}

docker/docker-compose.yaml:377

  • Same issue as above: env_file is added but the container vars are set via ${...} interpolation in environment (e.g., API_KEY: ${SANDBOX_API_KEY:-...}). env_file does not influence interpolation, so values in envs/core-services/sandbox.env won’t be used. Consider putting API_KEY, GIN_MODE, etc. directly into the sandbox env file and dropping the environment mappings, or document that these must come from the project .env instead.
    env_file:
      - ./envs/core-services/sandbox.env
      - ./envs/security.env
    environment:
      # The DifySandbox configurations
      # Make sure you are changing this key for your deployment with a strong key.
      # You can generate a strong key using `openssl rand -base64 42`.
      API_KEY: ${SANDBOX_API_KEY:-dify-sandbox}
      GIN_MODE: ${SANDBOX_GIN_MODE:-release}
      WORKER_TIMEOUT: ${SANDBOX_WORKER_TIMEOUT:-15}
      ENABLE_NETWORK: ${SANDBOX_ENABLE_NETWORK:-true}
      HTTP_PROXY: ${SANDBOX_HTTP_PROXY:-http://ssrf_proxy:3128}
      HTTPS_PROXY: ${SANDBOX_HTTPS_PROXY:-http://ssrf_proxy:3128}
      SANDBOX_PORT: ${SANDBOX_PORT:-8194}
      PIP_MIRROR_URL: ${PIP_MIRROR_URL:-}
    volumes:

docker/docker-compose.yaml:407

  • env_file is added for plugin_daemon, but the container values are still sourced via ${...} interpolation (e.g., SERVER_KEY: ${PLUGIN_DAEMON_KEY:-...}, DIFY_INNER_API_URL: ${PLUGIN_DIFY_INNER_API_URL:-...}). Since env_file doesn’t feed Compose interpolation and environment overrides env_file, changing envs/core-services/plugin-daemon.env won’t affect these settings. Either rely on env_file by defining SERVER_KEY/DIFY_INNER_API_URL directly there, or remove the env file entries and keep using .env interpolation consistently.
    env_file:
      - ./envs/core-services/shared.env
      - ./envs/core-services/plugin-daemon.env
      - ./envs/security.env
      - ./envs/databases/db-postgres.env
      - ./envs/databases/db-mysql.env
      - ./envs/databases/redis.env
    networks:
      - ssrf_proxy_network
      - default
    environment:
      DB_DATABASE: ${DB_PLUGIN_DATABASE:-dify_plugin}
      SERVER_PORT: ${PLUGIN_DAEMON_PORT:-5002}
      SERVER_KEY: ${PLUGIN_DAEMON_KEY:-lYkiYYT6owG+71oLerGzA7GXCgOT++6ovaezWAjpCjf+Sjc3ZtU+qUEi}
      MAX_PLUGIN_PACKAGE_SIZE: ${PLUGIN_MAX_PACKAGE_SIZE:-52428800}
      PPROF_ENABLED: ${PLUGIN_PPROF_ENABLED:-false}
      DIFY_INNER_API_URL: ${PLUGIN_DIFY_INNER_API_URL:-http://api:5001}
      DIFY_INNER_API_KEY: ${PLUGIN_DIFY_INNER_API_KEY:-QaHbTe77CtuXmsfyhR7+vRjI/+XbV1AaFy691iy+kGDv2Jvy0/eAh8Y1}
      PLUGIN_REMOTE_INSTALLING_HOST: ${PLUGIN_DEBUGGING_HOST:-0.0.0.0}

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +53 to +74
def create_env_files_from_example(env_example_path):
"""
Inserts the shared environment variables block and header comments into the template file,
removing any existing x-shared-env anchors, and generates the final docker-compose.yaml file.
Creates actual env files from .env.example by copying the categorized .env.example files.
This allows docker-compose to use env_file references.
Supports per-module structure with subdirectories.
"""
envs_dir = "envs"
os.makedirs(envs_dir, exist_ok=True)

created_files = []
# Walk through all .env.example files in subdirectories
for root, dirs, files in os.walk(envs_dir):
for file in files:
if file.endswith('.env.example'):
example_file = os.path.join(root, file)
env_file = example_file.replace('.env.example', '.env')

# Copy .example to actual file
with open(example_file, 'r') as src, open(env_file, 'w') as dst:
dst.write(src.read())
created_files.append(env_file)
print(f"Created {env_file}")
Copy link

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

create_env_files_from_example overwrites any existing .env files unconditionally. This can destroy user-customized env files when regenerating docker-compose. Only create the .env file if it doesn’t exist (or provide an explicit --force option), and consider copying docker/.env.example -> docker/.env here as well for parity with setup_env.sh.

Copilot uses AI. Check for mistakes.
Comment on lines +9 to +26
# Copy .env.example to .env in docker directory
if [ -f ".env.example" ]; then
if [ ! -f ".env" ]; then
cp .env.example .env
echo "Created .env from .env.example"
else
echo ".env already exists, skipping"
fi
fi

# Copy all .env.example files to .env files recursively
find envs -name "*.env.example" -type f | while read -r example_file; do
env_file="${example_file%.example}"
if [ ! -f "$env_file" ]; then
cp "$example_file" "$env_file"
echo "Created $env_file from $example_file"
fi
done
Copy link

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With set -e, find envs ... will cause the script to fail if envs/ doesn’t exist or if the script is invoked from a different working directory. Consider resolving paths relative to the script location (e.g., SCRIPT_DIR=...) and guard the find with a directory existence check.

Copilot uses AI. Check for mistakes.
Comment on lines 135 to 143

# API service
api:
<<: *shared-api-worker-config
image: langgenius/dify-api:1.11.4
restart: always
environment:
# Use the shared environment variables.
<<: *shared-api-worker-env
# Startup mode, 'api' starts the API server.
MODE: api
SENTRY_DSN: ${API_SENTRY_DSN:-}
Copy link

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

api now pulls values from env_file via the shared anchor, but the compose file still sets some related values using ${...} interpolation (e.g., SENTRY_DSN: ${API_SENTRY_DSN:-} here, and other similar entries below). Compose interpolation reads from the project .env/shell and does not use service env_file, so updating the split envs/**.env files won’t affect these interpolated settings. To make the split env files effective, prefer defining the container env vars directly in the env files (and removing the interpolated environment entries), or keep these vars in .env and don’t duplicate them in env_file.

Copilot uses AI. Check for mistakes.
ENABLE_OTEL=false
OTLP_TRACE_ENDPOINT=
OTLP_METRIC_ENDPOINT=
OTLP_BASE_ENDPOINT=http://localhost:4318# Prefix used to create collection name in vector database
Copy link

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line includes an inline # comment immediately after the URL, which most dotenv parsers will treat as part of the value (not a comment). That will make OTLP_BASE_ENDPOINT invalid. Move the comment to its own line (or add whitespace before # if your parser supports it).

Suggested change
OTLP_BASE_ENDPOINT=http://localhost:4318# Prefix used to create collection name in vector database
# Prefix used to create collection name in vector database
OTLP_BASE_ENDPOINT=http://localhost:4318

Copilot uses AI. Check for mistakes.
Comment on lines +53 to +60
def create_env_files_from_example(env_example_path):
"""
Inserts the shared environment variables block and header comments into the template file,
removing any existing x-shared-env anchors, and generates the final docker-compose.yaml file.
Creates actual env files from .env.example by copying the categorized .env.example files.
This allows docker-compose to use env_file references.
Supports per-module structure with subdirectories.
"""
envs_dir = "envs"
os.makedirs(envs_dir, exist_ok=True)
Copy link

Copilot AI Jan 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

env_example_path is unused in create_env_files_from_example, which makes the API misleading and hides the fact the function always operates on the hard-coded envs/ directory. Either use the passed-in path (e.g., to locate envs relative to it) or remove the parameter.

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

refactor size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Refactor/Chore] Split the environment configuration files to make docker compose clean again.

2 participants