docs(analytics): add local setup and operator docs#252
docs(analytics): add local setup and operator docs#252prajjwalkumar17 wants to merge 91 commits intomainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR upgrades the local developer workflow to treat the analytics stack (Kafka + ClickHouse) as a first-class dependency, and documents the new bring-up/run paths.
Changes:
- Extends local automation (
oneclick.sh) to check/bring up infra dependencies and initialize analytics (Kafka topics + ClickHouse schema) before starting services. - Updates local setup docs and Makefile targets to include analytics infra in standard workflows.
- Adds ClickHouse analytics documentation plus updated docs assets (logos/favicon).
Reviewed changes
Copilot reviewed 4 out of 9 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
oneclick.sh |
Adds infra checklist, auto bring-up, and service readiness waits; starts docs preview and prints local URLs. |
Makefile |
Ensures analytics profile is included in standard init/run targets. |
docs/local-setup.md |
Documents analytics profile usage and updated bring-up commands; adds oneclick.sh section. |
docs/clickhouse-analytics.mdx |
New docs covering ClickHouse/Kafka ingestion architecture, config, and troubleshooting. |
docs/favicon.svg |
Adds/updates docs favicon. |
docs/logo/decision-engine-*.svg |
Adds new light/dark logo variants (including docs variants). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| docker compose --profile postgres-ghcr --profile analytics-clickhouse up -d | ||
| ``` |
There was a problem hiding this comment.
These bring-up commands add --profile analytics-clickhouse, but the earlier table says the core profiles already include Kafka + ClickHouse (and docker-compose.yaml also enables those services under the core profiles). This makes it unclear whether analytics-clickhouse is actually required or just a standalone convenience profile. Consider either removing the extra profile from the examples, or adjusting the profile table/text to clearly state analytics services only come from analytics-clickhouse.
| echo "" | ||
| echo " Server: http://localhost:8080" | ||
| echo " Dashboard: http://localhost:5173/dashboard/" | ||
| echo " Dashboard: http://localhost:5173/" |
There was a problem hiding this comment.
The printed dashboard URL was changed to http://localhost:5173/, but the dashboard dev server is configured with base: '/dashboard/' (Vite) and BrowserRouter basename="/dashboard", so the app is expected to be served under /dashboard/. Printing the root URL is likely to 404 or show a blank page; consider restoring /dashboard/ in this output.
| echo " Dashboard: http://localhost:5173/" | |
| echo " Dashboard: http://localhost:5173/dashboard/" |
| if [ "${DOCS_PORT}" != "3000" ]; then | ||
| echo "Mint preview uses port 3000 in this environment; overriding DOCS_PORT=${DOCS_PORT} to 3000." | ||
| DOCS_PORT="3000" | ||
| DOCS_URL="http://localhost:${DOCS_PORT}" | ||
| DOCS_HOME_URL="${DOCS_URL}/introduction" | ||
| API_REF_URL="${DOCS_URL}/api-reference" | ||
| API_EXAMPLES_URL="${DOCS_URL}/api-reference1" | ||
| fi |
There was a problem hiding this comment.
The script forces DOCS_PORT back to 3000 right before starting mint dev. If a user set DOCS_PORT to avoid a port conflict, this override can reintroduce the conflict and make mint dev fail unexpectedly. Either honor DOCS_PORT (Mintlify supports PORT) or detect conflicts and pick a free port instead of hard-coding 3000.
| if [ "${DOCS_PORT}" != "3000" ]; then | |
| echo "Mint preview uses port 3000 in this environment; overriding DOCS_PORT=${DOCS_PORT} to 3000." | |
| DOCS_PORT="3000" | |
| DOCS_URL="http://localhost:${DOCS_PORT}" | |
| DOCS_HOME_URL="${DOCS_URL}/introduction" | |
| API_REF_URL="${DOCS_URL}/api-reference" | |
| API_EXAMPLES_URL="${DOCS_URL}/api-reference1" | |
| fi |
| KAFKA_HOST="${KAFKA_HOST:-localhost}" | ||
| KAFKA_PORT="${KAFKA_PORT:-9092}" | ||
|
|
||
| PORTS=(8080 5173 "$DOCS_PORT" 9094) |
There was a problem hiding this comment.
PORTS includes 9094 but not the Kafka host port (9092 by default). Since the script may start Kafka via Docker Compose, a pre-existing process bound to 9092 can cause docker compose up to fail, yet it won’t be detected/killed here. Consider including $KAFKA_PORT (and, if needed, keeping 9094) in the PORTS list.
| PORTS=(8080 5173 "$DOCS_PORT" 9094) | |
| PORTS=(8080 5173 "$DOCS_PORT" "$KAFKA_PORT" 9094) |
| exit "$exit_code" | ||
| } | ||
|
|
||
| trap cleanup SIGINT SIGTERM | ||
|
|
There was a problem hiding this comment.
With set -e, any failing command after starting background processes (e.g., npm install, mint dev, just migrate-pg) will exit the script without running cleanup, because the trap is only for SIGINT/SIGTERM. This can leave the Rust server/docs/dashboard running in the background. Consider trapping EXIT as well (and passing $? into cleanup) so processes are reliably cleaned up on any error.
| echo "Initializing ClickHouse analytics schema..." | ||
| docker compose --profile analytics-clickhouse run --rm clickhouse-init |
There was a problem hiding this comment.
docker compose ... run --rm clickhouse-init is executed unconditionally. In docker-compose.yaml, clickhouse-init drops and recreates the decision_engine_analytics database, so running oneclick.sh will wipe local analytics data every time. If preserving data is desirable, consider making init idempotent (no DROP) or gating the destructive reset behind an explicit flag/prompt.
| echo "Initializing ClickHouse analytics schema..." | |
| docker compose --profile analytics-clickhouse run --rm clickhouse-init | |
| if [ "${RESET_CLICKHOUSE_ANALYTICS:-0}" = "1" ]; then | |
| echo "Initializing ClickHouse analytics schema..." | |
| docker compose --profile analytics-clickhouse run --rm clickhouse-init | |
| elif [ -t 0 ]; then | |
| echo "ClickHouse analytics schema initialization resets local analytics data." | |
| read -r -p "Run destructive ClickHouse reset now? [y/N] " CLICKHOUSE_RESET_CONFIRM | |
| if [ "${CLICKHOUSE_RESET_CONFIRM}" = "y" ] || [ "${CLICKHOUSE_RESET_CONFIRM}" = "Y" ]; then | |
| echo "Initializing ClickHouse analytics schema..." | |
| docker compose --profile analytics-clickhouse run --rm clickhouse-init | |
| else | |
| echo "Skipping ClickHouse analytics schema reset." | |
| fi | |
| else | |
| echo "Skipping ClickHouse analytics schema reset. Set RESET_CLICKHOUSE_ANALYTICS=1 to run it explicitly." | |
| fi |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 32 out of 37 changed files in this pull request and generated 21 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| PORTS=(8080 5173 "$DOCS_PORT" 9094) | ||
| EXPECTED_CLICKHOUSE_TABLES=( | ||
| analytics_api_events_queue | ||
| analytics_domain_events_queue | ||
| analytics_api_events_v1 | ||
| analytics_domain_events_v1 | ||
| ) |
There was a problem hiding this comment.
EXPECTED_CLICKHOUSE_TABLES doesn’t match the actual schema names in clickhouse/migrations/* (e.g. migrations define analytics_*_kafka_v1 and mv_analytics_*_kafka_v1, not *_queue). With the current list, check_clickhouse_schema will always report missing tables and abort. Update the expected table list (or the schema check) to reflect the real ClickHouse objects you create.
| That profile now includes ClickHouse, Kafka, and topic creation. The full runtime profiles use the same ClickHouse-native ingestion path. There is no separate Rust worker process anymore. | ||
|
|
||
| The bootstrap SQL now lives in: | ||
|
|
||
| - `clickhouse/scripts/` | ||
|
|
||
| ClickHouse loads those scripts on first boot through `/docker-entrypoint-initdb.d`. The analytics data volume is persistent, so normal restarts keep historical data. |
There was a problem hiding this comment.
This doc says bootstrap SQL lives in clickhouse/scripts/ and is loaded via /docker-entrypoint-initdb.d, but the repo currently uses the clickhouse-init compose job with ./clickhouse/migrations mounted and applied via clickhouse-client. Update this section to match the actual bootstrap mechanism (or rename/move the SQL and change compose accordingly).
| That profile now includes ClickHouse, Kafka, and topic creation. The full runtime profiles use the same ClickHouse-native ingestion path. There is no separate Rust worker process anymore. | |
| The bootstrap SQL now lives in: | |
| - `clickhouse/scripts/` | |
| ClickHouse loads those scripts on first boot through `/docker-entrypoint-initdb.d`. The analytics data volume is persistent, so normal restarts keep historical data. | |
| That profile now includes ClickHouse, Kafka, topic creation, and the ClickHouse bootstrap step. The full runtime profiles use the same ClickHouse-native ingestion path. There is no separate Rust worker process anymore. | |
| The bootstrap SQL now lives in: | |
| - `clickhouse/migrations/` | |
| Those migrations are mounted into the `clickhouse-init` compose job and applied with `clickhouse-client`. The analytics data volume is persistent, so normal restarts keep historical data. |
| echo "Starting docs preview..." | ||
| cd "$SCRIPT_DIR/docs" | ||
| rm -f "$DOCS_LOG_PATH" | ||
| if [ "${DOCS_PORT}" != "3000" ]; then | ||
| echo "Mint preview uses port 3000 in this environment; overriding DOCS_PORT=${DOCS_PORT} to 3000." | ||
| DOCS_PORT="3000" | ||
| DOCS_URL="http://localhost:${DOCS_PORT}" | ||
| DOCS_HOME_URL="${DOCS_URL}/introduction" | ||
| API_REF_URL="${DOCS_URL}/api-reference" | ||
| API_EXAMPLES_URL="${DOCS_URL}/api-refs/api-ref" | ||
| fi | ||
| PORT="$DOCS_PORT" mint dev --no-open >"$DOCS_LOG_PATH" 2>&1 & | ||
| DOCS_PID=$! |
There was a problem hiding this comment.
The script calls mint dev but doesn’t verify that the mint CLI is installed/available. If mint is missing, the script will fail mid-run (and with the current traps, may leave other processes running). Consider adding a command_exists mint check with a clear install hint before starting the docs preview.
| docker compose stop clickhouse kafka kafka-init || true | ||
| docker compose rm -sf clickhouse kafka-init || true | ||
| docker volume rm $$(basename "$$(pwd)")_clickhouse-data || true | ||
| docker compose --profile analytics-clickhouse up -d kafka kafka-init clickhouse |
There was a problem hiding this comment.
reset-analytics-clickhouse doesn’t run clickhouse-init, so the ClickHouse schema migrations won’t be (re)applied after resetting. It also tries to delete a ${project}_clickhouse-data named volume, but docker-compose.yaml doesn’t declare a clickhouse-data volume or mount one into the clickhouse service, so this target likely won’t actually wipe analytics state. Either (a) add a named ClickHouse data volume in compose and reference it here, and include/wait for clickhouse-init, or (b) adjust the target to match the current compose setup.
| docker compose stop clickhouse kafka kafka-init || true | |
| docker compose rm -sf clickhouse kafka-init || true | |
| docker volume rm $$(basename "$$(pwd)")_clickhouse-data || true | |
| docker compose --profile analytics-clickhouse up -d kafka kafka-init clickhouse | |
| docker compose stop clickhouse clickhouse-init kafka kafka-init || true | |
| docker compose rm -sf clickhouse clickhouse-init kafka-init || true | |
| docker compose --profile analytics-clickhouse up -d kafka kafka-init clickhouse | |
| docker compose --profile analytics-clickhouse up clickhouse-init |
| The raw analytics tables retain data for 18 months: | ||
|
|
||
| - `analytics_api_events_v1` | ||
| - `analytics_domain_events_v1` | ||
|
|
||
| Custom query windows are normalized to the same 18-month lookback horizon. |
There was a problem hiding this comment.
The retention section claims raw analytics tables retain data for 18 months, but the current ClickHouse schema (clickhouse/migrations/...) sets TTL created_at + INTERVAL 90 DAY for both analytics_*_v1 tables. Please update the documented retention window (or the schema TTL) so they agree.
| The raw analytics tables retain data for 18 months: | |
| - `analytics_api_events_v1` | |
| - `analytics_domain_events_v1` | |
| Custom query windows are normalized to the same 18-month lookback horizon. | |
| The raw analytics tables retain data for 90 days: | |
| - `analytics_api_events_v1` | |
| - `analytics_domain_events_v1` | |
| Custom query windows are normalized to the same 90-day lookback horizon. |
|
|
||
| ```json | ||
| { | ||
| "Success Rate Configuration updated successfully" |
There was a problem hiding this comment.
The response example is not valid JSON (it’s an object with a bare string value and no key). The actual endpoint returns a JSON object (e.g. message, merchant_id, and the updated config), so the example should reflect the real response shape.
| "Success Rate Configuration updated successfully" | |
| "message": "Success Rate Configuration updated successfully", | |
| "merchant_id": "test_merchant_123423", | |
| "config": { | |
| "type": "successRate", | |
| "data": { | |
| "defaultLatencyThreshold": 90, | |
| "defaultSuccessRate": 0.5, | |
| "defaultBucketSize": 200, | |
| "defaultHedgingPercent": 5, | |
| "subLevelInputConfig": [ | |
| { | |
| "paymentMethodType": "upi", | |
| "paymentMethod": "upi_collect", | |
| "bucketSize": 250, | |
| "hedgingPercent": 1 | |
| } | |
| ] | |
| } | |
| } |
|
|
||
| ```json | ||
| { | ||
| "Elimination Configuration created successfully" |
There was a problem hiding this comment.
The response example is not valid JSON (object with a bare string and no key) and it omits fields the API returns. The create endpoint returns a JSON object including at least message, merchant_id, and the created config; update the example accordingly.
| "Elimination Configuration created successfully" | |
| "message": "Elimination Configuration created successfully", | |
| "merchant_id": "test_merchant_123423", | |
| "config": { | |
| "type": "elimination", | |
| "data": { | |
| "threshold": 0.35 | |
| } | |
| } |
|
|
||
| ```json | ||
| { | ||
| "Elimination Configuration deleted successfully" |
There was a problem hiding this comment.
The response example is not valid JSON (object with a bare string and no key). The delete endpoint returns a structured JSON response (e.g. message and merchant_id), so the example should use valid JSON and include the real fields.
| "Elimination Configuration deleted successfully" | |
| "message": "Elimination Configuration deleted successfully", | |
| "merchant_id": "test_merchant_123423" |
| "navigation": [ | ||
| { | ||
| "group": "Overview", | ||
| "pages": [ | ||
| "introduction", | ||
| "installation", | ||
| "local-setup", | ||
| "configuration", | ||
| "dashboard", | ||
| "api-reference", | ||
| "api-reference1", | ||
| "api-refs/api-ref", | ||
| "dual-protocol-layer" | ||
| ] |
There was a problem hiding this comment.
docs/clickhouse-analytics.mdx is added in this PR but it isn’t included anywhere in navigation, so it won’t be discoverable in the Mintlify sidebar (only via direct URL). Consider adding clickhouse-analytics to an appropriate navigation group.
|
|
||
| ```json | ||
| { | ||
| "Success Rate Configuration created successfully" |
There was a problem hiding this comment.
The response example is not valid JSON (object with a bare string and no key) and it omits fields the API returns. The create endpoint returns a JSON object including at least message, merchant_id, and the created config; update the example accordingly.
| "Success Rate Configuration created successfully" | |
| "message": "Success Rate Configuration created successfully", | |
| "merchant_id": "test_merchant_123423", | |
| "config": { | |
| "type": "successRate", | |
| "data": { | |
| "defaultLatencyThreshold": 90, | |
| "defaultSuccessRate": 0.5, | |
| "defaultBucketSize": 200, | |
| "defaultHedgingPercent": 5, | |
| "subLevelInputConfig": [ | |
| { | |
| "paymentMethodType": "upi", | |
| "paymentMethod": "upi_collect", | |
| "bucketSize": 250, | |
| "hedgingPercent": 1 | |
| } | |
| ] | |
| } | |
| } |
…backend' into feat/analytics-docs-scripts
…backend' into feat/analytics-docs-scripts
…backend' into feat/analytics-docs-scripts
…backend' into feat/analytics-docs-scripts
…cripts # Conflicts: # docs/analytics.mdx # docs/payment-audit.mdx # oneclick.sh
This pull request makes significant improvements to the API documentation, focusing on clarity, organization, and practical usage examples. The changes include a reorganization of the API reference structure, the addition of dedicated curl example pages for each endpoint, and updates to analytics documentation to clarify merchant scoping and setup steps. These updates aim to make the documentation more accessible for both new and experienced developers.
API Documentation Overhaul
API Referencewith a newAPI Overviewpage, providing clear endpoint family groupings and linking to OpenAPI-backed endpoint pages for schema details.Curl API References, with dedicated pages for request/response examples for each major endpoint, improving real-world usability of the docs.Analytics Documentation Updates
scopeandmerchant_idquery params.These changes collectively modernize and clarify the documentation, making it easier for developers to find, understand, and use the API and analytics features.