-
Notifications
You must be signed in to change notification settings - Fork 19.9k
refactor: split docker-compose env config into separate files #31586
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
refactor: split docker-compose env config into separate files #31586
Conversation
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
- Split environment variables into categorized files (global, server, security, database, redis, storage, vectorstore) - Replace large YAML anchor with env_file references using x-shared-config - Update generate_docker_compose script to create env files from examples - Makes docker-compose.yaml cleaner and more maintainable Fixes langgenius#31572
d79644e to
0882f1e
Compare
- Update test to read from env files instead of x-shared-env anchor - Matches the new docker-compose structure using env_file
- Reorganize into: core-services, security, databases, vectorstores, infrastructure - Core Services: api, worker, worker_beat, web, sandbox, plugin_daemon - Databases: db_postgres, db_mysql, redis - Vector Stores: all vector store services - Infrastructure: nginx, certbot, ssrf_proxy, etc. - Security: general security configurations Updates langgenius#31572
0468a18 to
c28eb0f
Compare
- Split into per-module structure: core-services/api, core-services/worker, etc. - Core services: shared.env (for api/worker/worker-beat), api.env, worker.env, worker-beat.env, web.env, sandbox.env, plugin-daemon.env - Databases: db-postgres.env, db-mysql.env, redis.env - Vector stores: separate file per vector store service - Infrastructure: separate file per infrastructure service - Update docker-compose-template.yaml to reference per-module files - Update generate_docker_compose and test to handle recursive structure Updates langgenius#31572
- Remove core-services.env, databases.env, infrastructure.env, vectorstores.env - Keep security.env as it's still referenced in docker-compose - All variables are now in per-module structure
- Add VECTOR_STORE, WEAVIATE_*, QDRANT_*, OCEANBASE_*, PGVECTOR_*, MILVUS_* variables - These are used by api/worker services and should be in shared.env - Fixes pytest config test failure
crazywoola
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI. These *.env should not be committed into the main branch. Only *.example should be added.
In order to keep the start command clean. I think we need a way to simplify the command as well.
cd dify
cd docker
cp .env.example .env <= This line
docker compose up -d
- Remove all .env files from git (keep only .env.example files) - Add .gitignore to ignore .env files - Add setup_env.sh script to copy all .env.example files to .env - Simplifies setup: users run './setup_env.sh' instead of copying many files manually
- Replace manual 'cp .env.example .env' with './setup_env.sh' - Simplifies setup for users with per-module env file structure
- generate_docker_compose already creates .env files from .env.example - This ensures .env files exist in CI/CD environments - Fixes docker-compose error about missing env files
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
Refactors the Docker Compose environment configuration by splitting the previous large shared YAML environment anchor into multiple categorized envs/**/*.env(.example) files and updating generation/setup workflows accordingly (Fixes #31572).
Changes:
- Introduces categorized
docker/envs/**.env.examplefiles and updates Compose to load them viaenv_file. - Updates
docker/generate_docker_composeand addsdocker/setup_env.shto generate/copy.envfiles from examples. - Adjusts config parity test to read the new split env files instead of the removed shared YAML anchor.
Reviewed changes
Copilot reviewed 41 out of 41 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| docker/setup_env.sh | Adds setup script to copy .env.example → .env (and envs examples → envs envs). |
| docker/generate_docker_compose | Stops injecting shared YAML env anchor; adds env-file generation from examples. |
| docker/envs/security.env.example | New split security env example file. |
| docker/envs/core-services/api.env.example | New API-specific env example. |
| docker/envs/core-services/plugin-daemon.env.example | New plugin-daemon env example. |
| docker/envs/core-services/sandbox.env.example | New sandbox env example. |
| docker/envs/core-services/shared.env.example | New shared API/worker env example (large split from old anchor). |
| docker/envs/core-services/web.env.example | New web env example. |
| docker/envs/core-services/worker-beat.env.example | New worker-beat env example. |
| docker/envs/core-services/worker.env.example | New worker env example. |
| docker/envs/databases/db-mysql.env.example | New MySQL env example. |
| docker/envs/databases/db-postgres.env.example | New Postgres env example. |
| docker/envs/databases/redis.env.example | New Redis env example. |
| docker/envs/infrastructure/certbot.env.example | New certbot env example. |
| docker/envs/infrastructure/etcd.env.example | New etcd env example. |
| docker/envs/infrastructure/milvus-standalone.env.example | New milvus-standalone env example. |
| docker/envs/infrastructure/minio.env.example | New minio env example. |
| docker/envs/infrastructure/nginx.env.example | New nginx env example. |
| docker/envs/infrastructure/ssrf-proxy.env.example | New ssrf-proxy env example. |
| docker/envs/vectorstores/chroma.env.example | New Chroma env example. |
| docker/envs/vectorstores/couchbase.env.example | New Couchbase env example. |
| docker/envs/vectorstores/elasticsearch.env.example | New Elasticsearch env example. |
| docker/envs/vectorstores/iris.env.example | New Iris env example. |
| docker/envs/vectorstores/matrixone.env.example | New MatrixOne env example. |
| docker/envs/vectorstores/milvus.env.example | New Milvus env example. |
| docker/envs/vectorstores/myscale.env.example | New MyScale env example. |
| docker/envs/vectorstores/oceanbase.env.example | New OceanBase env example. |
| docker/envs/vectorstores/opengauss.env.example | New OpenGauss env example. |
| docker/envs/vectorstores/opensearch.env.example | New OpenSearch env example. |
| docker/envs/vectorstores/oracle.env.example | New Oracle env example. |
| docker/envs/vectorstores/pgvecto-rs.env.example | New pgvecto-rs env example. |
| docker/envs/vectorstores/pgvector.env.example | New pgvector env example. |
| docker/envs/vectorstores/qdrant.env.example | New Qdrant env example. |
| docker/envs/vectorstores/seekdb.env.example | New SeekDB env example. |
| docker/envs/vectorstores/vastbase.env.example | New Vastbase env example. |
| docker/envs/vectorstores/weaviate.env.example | New Weaviate env example. |
| docker/docker-compose.yaml | Switches to shared anchor(s) that use env_file instead of huge YAML env anchor. |
| docker/docker-compose-template.yaml | Updates template to match new env_file-based configuration. |
| docker/.gitignore | Ignores generated .env files while keeping .env.example tracked. |
| dev/pytest/pytest_config_tests.py | Updates docker-compose config-key discovery to read split env example files. |
| README.md | Updates Docker Compose setup instructions to use ./setup_env.sh. |
Comments suppressed due to low confidence (7)
docker/docker-compose-template.yaml:247
env_fileis added, but core settings are also set underenvironmentusing${...}(e.g.,CONSOLE_API_URL,APP_API_URL). Sinceenv_filedoesn’t feed Compose interpolation andenvironmentoverridesenv_file, changes toenvs/core-services/web.envwon’t affect these values. To make the split env files effective, remove the duplicatedenvironmententries or define the exact container env vars in the env file and rely onenv_fileonly.
env_file:
- ./envs/core-services/web.env
- ./envs/security.env
environment:
CONSOLE_API_URL: ${CONSOLE_API_URL:-}
APP_API_URL: ${APP_API_URL:-}
AMPLITUDE_API_KEY: ${AMPLITUDE_API_KEY:-}
NEXT_PUBLIC_COOKIE_DOMAIN: ${NEXT_PUBLIC_COOKIE_DOMAIN:-}
SENTRY_DSN: ${WEB_SENTRY_DSN:-}
docker/docker-compose-template.yaml:144
- The template introduces
env_filefor shared config, but this service still sets many values via${...}interpolation underenvironment(e.g.,PLUGIN_DAEMON_TIMEOUT,INNER_API_KEY_FOR_PLUGIN). Compose interpolation reads from the project.env/shell and does not use serviceenv_file, and any keys repeated underenvironmentoverrideenv_file. As a result, editingenvs/**.envwon’t actually change these settings. Prefer defining the final container env var names directly in the env files and removing the duplicatedenvironmententries, or keep these vars in.envand don’t include them inenv_file.
<<: *shared-api-worker-config
image: langgenius/dify-api:1.11.4
environment:
# Startup mode, 'api' starts the API server.
MODE: api
SENTRY_DSN: ${API_SENTRY_DSN:-}
SENTRY_TRACES_SAMPLE_RATE: ${API_SENTRY_TRACES_SAMPLE_RATE:-1.0}
SENTRY_PROFILES_SAMPLE_RATE: ${API_SENTRY_PROFILES_SAMPLE_RATE:-1.0}
PLUGIN_REMOTE_INSTALL_HOST: ${EXPOSE_PLUGIN_DEBUGGING_HOST:-localhost}
PLUGIN_REMOTE_INSTALL_PORT: ${EXPOSE_PLUGIN_DEBUGGING_PORT:-5003}
PLUGIN_MAX_PACKAGE_SIZE: ${PLUGIN_MAX_PACKAGE_SIZE:-52428800}
PLUGIN_DAEMON_TIMEOUT: ${PLUGIN_DAEMON_TIMEOUT:-600.0}
INNER_API_KEY_FOR_PLUGIN: ${PLUGIN_DIFY_INNER_API_KEY:-QaHbTe77CtuXmsfyhR7+vRjI/+XbV1AaFy691iy+kGDv2Jvy0/eAh8Y1}
docker/docker-compose-template.yaml:369
- Same problem here:
env_fileis added but the container vars are set via${...}interpolation (e.g.,API_KEY: ${SANDBOX_API_KEY:-...}), which won’t pick up values fromenvs/core-services/sandbox.env. Consider puttingAPI_KEY,GIN_MODE, etc. directly into the sandbox env file and dropping theenvironmentmappings, or document that these must come from the project.envinstead.
env_file:
- ./envs/core-services/sandbox.env
- ./envs/security.env
environment:
# The DifySandbox configurations
# Make sure you are changing this key for your deployment with a strong key.
# You can generate a strong key using `openssl rand -base64 42`.
API_KEY: ${SANDBOX_API_KEY:-dify-sandbox}
GIN_MODE: ${SANDBOX_GIN_MODE:-release}
WORKER_TIMEOUT: ${SANDBOX_WORKER_TIMEOUT:-15}
ENABLE_NETWORK: ${SANDBOX_ENABLE_NETWORK:-true}
HTTP_PROXY: ${SANDBOX_HTTP_PROXY:-http://ssrf_proxy:3128}
HTTPS_PROXY: ${SANDBOX_HTTPS_PROXY:-http://ssrf_proxy:3128}
SANDBOX_PORT: ${SANDBOX_PORT:-8194}
docker/docker-compose-template.yaml:400
env_fileis added for plugin_daemon, but critical values are still sourced via${...}interpolation (e.g.,SERVER_KEY,DIFY_INNER_API_URL). Sinceenv_filedoesn’t influence interpolation andenvironmentoverridesenv_file, updatingenvs/core-services/plugin-daemon.envwon’t affect these settings. Either rely onenv_fileby definingSERVER_KEY/DIFY_INNER_API_URLdirectly there, or remove the env file entries and keep using.envinterpolation consistently.
env_file:
- ./envs/core-services/shared.env
- ./envs/core-services/plugin-daemon.env
- ./envs/security.env
- ./envs/databases/db-postgres.env
- ./envs/databases/db-mysql.env
- ./envs/databases/redis.env
networks:
- ssrf_proxy_network
- default
environment:
DB_DATABASE: ${DB_PLUGIN_DATABASE:-dify_plugin}
SERVER_PORT: ${PLUGIN_DAEMON_PORT:-5002}
SERVER_KEY: ${PLUGIN_DAEMON_KEY:-lYkiYYT6owG+71oLerGzA7GXCgOT++6ovaezWAjpCjf+Sjc3ZtU+qUEi}
MAX_PLUGIN_PACKAGE_SIZE: ${PLUGIN_MAX_PACKAGE_SIZE:-52428800}
PPROF_ENABLED: ${PLUGIN_PPROF_ENABLED:-false}
DIFY_INNER_API_URL: ${PLUGIN_DIFY_INNER_API_URL:-http://api:5001}
DIFY_INNER_API_KEY: ${PLUGIN_DIFY_INNER_API_KEY:-QaHbTe77CtuXmsfyhR7+vRjI/+XbV1AaFy691iy+kGDv2Jvy0/eAh8Y1}
docker/docker-compose.yaml:252
env_fileis added, but the same values are also set inenvironmentusing${...}interpolation (e.g.,CONSOLE_API_URL,APP_API_URL). Compose interpolation reads from the project.env/shell, not from serviceenv_file, andenvironmentoverridesenv_file, so changes toenvs/core-services/web.envwon’t affect these variables. To make the split env files effective, remove these duplicatedenvironmententries (or change the env file to define the exact container env var names and rely onenv_fileonly).
env_file:
- ./envs/core-services/web.env
- ./envs/security.env
environment:
CONSOLE_API_URL: ${CONSOLE_API_URL:-}
APP_API_URL: ${APP_API_URL:-}
AMPLITUDE_API_KEY: ${AMPLITUDE_API_KEY:-}
NEXT_PUBLIC_COOKIE_DOMAIN: ${NEXT_PUBLIC_COOKIE_DOMAIN:-}
docker/docker-compose.yaml:377
- Same issue as above:
env_fileis added but the container vars are set via${...}interpolation inenvironment(e.g.,API_KEY: ${SANDBOX_API_KEY:-...}).env_filedoes not influence interpolation, so values inenvs/core-services/sandbox.envwon’t be used. Consider puttingAPI_KEY,GIN_MODE, etc. directly into the sandbox env file and dropping theenvironmentmappings, or document that these must come from the project.envinstead.
env_file:
- ./envs/core-services/sandbox.env
- ./envs/security.env
environment:
# The DifySandbox configurations
# Make sure you are changing this key for your deployment with a strong key.
# You can generate a strong key using `openssl rand -base64 42`.
API_KEY: ${SANDBOX_API_KEY:-dify-sandbox}
GIN_MODE: ${SANDBOX_GIN_MODE:-release}
WORKER_TIMEOUT: ${SANDBOX_WORKER_TIMEOUT:-15}
ENABLE_NETWORK: ${SANDBOX_ENABLE_NETWORK:-true}
HTTP_PROXY: ${SANDBOX_HTTP_PROXY:-http://ssrf_proxy:3128}
HTTPS_PROXY: ${SANDBOX_HTTPS_PROXY:-http://ssrf_proxy:3128}
SANDBOX_PORT: ${SANDBOX_PORT:-8194}
PIP_MIRROR_URL: ${PIP_MIRROR_URL:-}
volumes:
docker/docker-compose.yaml:407
env_fileis added for plugin_daemon, but the container values are still sourced via${...}interpolation (e.g.,SERVER_KEY: ${PLUGIN_DAEMON_KEY:-...},DIFY_INNER_API_URL: ${PLUGIN_DIFY_INNER_API_URL:-...}). Sinceenv_filedoesn’t feed Compose interpolation andenvironmentoverridesenv_file, changingenvs/core-services/plugin-daemon.envwon’t affect these settings. Either rely onenv_fileby definingSERVER_KEY/DIFY_INNER_API_URLdirectly there, or remove the env file entries and keep using.envinterpolation consistently.
env_file:
- ./envs/core-services/shared.env
- ./envs/core-services/plugin-daemon.env
- ./envs/security.env
- ./envs/databases/db-postgres.env
- ./envs/databases/db-mysql.env
- ./envs/databases/redis.env
networks:
- ssrf_proxy_network
- default
environment:
DB_DATABASE: ${DB_PLUGIN_DATABASE:-dify_plugin}
SERVER_PORT: ${PLUGIN_DAEMON_PORT:-5002}
SERVER_KEY: ${PLUGIN_DAEMON_KEY:-lYkiYYT6owG+71oLerGzA7GXCgOT++6ovaezWAjpCjf+Sjc3ZtU+qUEi}
MAX_PLUGIN_PACKAGE_SIZE: ${PLUGIN_MAX_PACKAGE_SIZE:-52428800}
PPROF_ENABLED: ${PLUGIN_PPROF_ENABLED:-false}
DIFY_INNER_API_URL: ${PLUGIN_DIFY_INNER_API_URL:-http://api:5001}
DIFY_INNER_API_KEY: ${PLUGIN_DIFY_INNER_API_KEY:-QaHbTe77CtuXmsfyhR7+vRjI/+XbV1AaFy691iy+kGDv2Jvy0/eAh8Y1}
PLUGIN_REMOTE_INSTALLING_HOST: ${PLUGIN_DEBUGGING_HOST:-0.0.0.0}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| def create_env_files_from_example(env_example_path): | ||
| """ | ||
| Inserts the shared environment variables block and header comments into the template file, | ||
| removing any existing x-shared-env anchors, and generates the final docker-compose.yaml file. | ||
| Creates actual env files from .env.example by copying the categorized .env.example files. | ||
| This allows docker-compose to use env_file references. | ||
| Supports per-module structure with subdirectories. | ||
| """ | ||
| envs_dir = "envs" | ||
| os.makedirs(envs_dir, exist_ok=True) | ||
|
|
||
| created_files = [] | ||
| # Walk through all .env.example files in subdirectories | ||
| for root, dirs, files in os.walk(envs_dir): | ||
| for file in files: | ||
| if file.endswith('.env.example'): | ||
| example_file = os.path.join(root, file) | ||
| env_file = example_file.replace('.env.example', '.env') | ||
|
|
||
| # Copy .example to actual file | ||
| with open(example_file, 'r') as src, open(env_file, 'w') as dst: | ||
| dst.write(src.read()) | ||
| created_files.append(env_file) | ||
| print(f"Created {env_file}") |
Copilot
AI
Jan 27, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
create_env_files_from_example overwrites any existing .env files unconditionally. This can destroy user-customized env files when regenerating docker-compose. Only create the .env file if it doesn’t exist (or provide an explicit --force option), and consider copying docker/.env.example -> docker/.env here as well for parity with setup_env.sh.
| # Copy .env.example to .env in docker directory | ||
| if [ -f ".env.example" ]; then | ||
| if [ ! -f ".env" ]; then | ||
| cp .env.example .env | ||
| echo "Created .env from .env.example" | ||
| else | ||
| echo ".env already exists, skipping" | ||
| fi | ||
| fi | ||
|
|
||
| # Copy all .env.example files to .env files recursively | ||
| find envs -name "*.env.example" -type f | while read -r example_file; do | ||
| env_file="${example_file%.example}" | ||
| if [ ! -f "$env_file" ]; then | ||
| cp "$example_file" "$env_file" | ||
| echo "Created $env_file from $example_file" | ||
| fi | ||
| done |
Copilot
AI
Jan 27, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With set -e, find envs ... will cause the script to fail if envs/ doesn’t exist or if the script is invoked from a different working directory. Consider resolving paths relative to the script location (e.g., SCRIPT_DIR=...) and guard the find with a directory existence check.
|
|
||
| # API service | ||
| api: | ||
| <<: *shared-api-worker-config | ||
| image: langgenius/dify-api:1.11.4 | ||
| restart: always | ||
| environment: | ||
| # Use the shared environment variables. | ||
| <<: *shared-api-worker-env | ||
| # Startup mode, 'api' starts the API server. | ||
| MODE: api | ||
| SENTRY_DSN: ${API_SENTRY_DSN:-} |
Copilot
AI
Jan 27, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
api now pulls values from env_file via the shared anchor, but the compose file still sets some related values using ${...} interpolation (e.g., SENTRY_DSN: ${API_SENTRY_DSN:-} here, and other similar entries below). Compose interpolation reads from the project .env/shell and does not use service env_file, so updating the split envs/**.env files won’t affect these interpolated settings. To make the split env files effective, prefer defining the container env vars directly in the env files (and removing the interpolated environment entries), or keep these vars in .env and don’t duplicate them in env_file.
| ENABLE_OTEL=false | ||
| OTLP_TRACE_ENDPOINT= | ||
| OTLP_METRIC_ENDPOINT= | ||
| OTLP_BASE_ENDPOINT=http://localhost:4318# Prefix used to create collection name in vector database |
Copilot
AI
Jan 27, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line includes an inline # comment immediately after the URL, which most dotenv parsers will treat as part of the value (not a comment). That will make OTLP_BASE_ENDPOINT invalid. Move the comment to its own line (or add whitespace before # if your parser supports it).
| OTLP_BASE_ENDPOINT=http://localhost:4318# Prefix used to create collection name in vector database | |
| # Prefix used to create collection name in vector database | |
| OTLP_BASE_ENDPOINT=http://localhost:4318 |
| def create_env_files_from_example(env_example_path): | ||
| """ | ||
| Inserts the shared environment variables block and header comments into the template file, | ||
| removing any existing x-shared-env anchors, and generates the final docker-compose.yaml file. | ||
| Creates actual env files from .env.example by copying the categorized .env.example files. | ||
| This allows docker-compose to use env_file references. | ||
| Supports per-module structure with subdirectories. | ||
| """ | ||
| envs_dir = "envs" | ||
| os.makedirs(envs_dir, exist_ok=True) |
Copilot
AI
Jan 27, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
env_example_path is unused in create_env_files_from_example, which makes the API misleading and hides the fact the function always operates on the hard-coded envs/ directory. Either use the passed-in path (e.g., to locate envs relative to it) or remove the parameter.
Co-authored-by: Copilot <[email protected]>
Refactors docker-compose configuration to split environment variables into separate, organized files.
Changes:
Fixes #31572