-
Notifications
You must be signed in to change notification settings - Fork 175
Add Temporal Cloud OpenMetrics Prometheus+Grafana demo with scenario workflows #766
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
deepika-awasthi
wants to merge
2
commits into
main
Choose a base branch
from
deepika/openmetrics-demo
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
102 changes: 102 additions & 0 deletions
102
core/src/main/java/io/temporal/samples/temporalcloudopenmetrics/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| # Temporal Cloud OpenMetrics → Prometheus → Grafana (Step-by-step) | ||
|
|
||
| This demo shows how to **scrape Temporal Cloud OpenMetrics(https://docs.temporal.io/cloud/metrics/openmetrics/)** | ||
| with **Prometheus** and **visualize them in Grafana**. | ||
|
|
||
| It uses the Grafana Temporal mixin dashboard template: | ||
| https://github.com/grafana/jsonnet-libs/blob/master/temporal-mixin/dashboards/temporal-overview.json | ||
|
|
||
| Once imported/provisioned, the dashboard lets you view the key Temporal metrics in a ready-made layout. | ||
|
|
||
| Grafana dashboard view :- | ||
|  | ||
|  | ||
|
|
||
| Prometheus - | ||
|
|
||
|  | ||
|
|
||
| --- | ||
|
|
||
| ## 1) Create Service Account + API Key (Temporal Cloud) | ||
|
|
||
| OpenMetrics auth reference: | ||
| https://docs.temporal.io/production-deployment/cloud/metrics/openmetrics/api-reference#authentication | ||
|
|
||
| In Temporal Cloud UI: | ||
| - **Settings → Service Accounts** | ||
| - Create a service account with **Metrics Read-Only** role | ||
| - Generate an **API key** ( copy this, it will be needed later) | ||
| --- | ||
|
|
||
|
|
||
| ## 2) Create the `secrets/` folder + API key file | ||
|
|
||
| From the repo root (same folder as `docker-compose.yml`), run: | ||
|
|
||
| ``` | ||
| cd temporalcloudopenmetrics | ||
| mkdir -p secrets | ||
| echo "put your CLOUD API KEY HERE" > secrets/temporal_cloud_api_key | ||
| ``` | ||
| now the folder will look like below and temporal_cloud_api_key will have the above api key that we generated in step 1 | ||
|
|
||
| ``` | ||
| temporalcloudopenmetrics/ | ||
| ├── docker-compose.yml | ||
| ├── prometheus/ | ||
| ├── grafana/ | ||
| ├── secrets/ | ||
| │ └── temporal_cloud_api_key | ||
| ``` | ||
|
|
||
| ## 3) Configure TemporalConnection.java | ||
|
|
||
| Edit `TemporalConnection.java` and set your defaults: | ||
|
|
||
| ``` | ||
| public static final String NAMESPACE = env("TEMPORAL_NAMESPACE", "<namespace>.<account-id>"); | ||
| public static final String ADDRESS = env("TEMPORAL_ADDRESS", "<namespace>.<account-id>.tmprl.cloud:7233"); | ||
| public static final String CERT = env("TEMPORAL_CERT", "/path/to/client.pem"); | ||
| public static final String KEY = env("TEMPORAL_KEY", "/path/to/client.key"); | ||
| public static final String TASK_QUEUE = env("TASK_QUEUE", "openmetrics-task-queue"); | ||
| public static final int WORKER_SECONDS = envInt("WORKER_SECONDS", 60); | ||
| ``` | ||
|
|
||
| ## 4) 4) Update Prometheus scrape config | ||
|
|
||
| prometheus/config.yml | ||
| Update it to use your namespace | ||
| ``` | ||
| params: | ||
| namespaces: [ '<namespace>.<account-id>' ] | ||
| ``` | ||
|
|
||
|
|
||
| ## 5) Start Prometheus + Grafana | ||
|
|
||
| docker compose up -d | ||
| docker compose ps | ||
|
|
||
|
|
||
| ## 6) View Grafana dashboard | ||
|
|
||
| http://localhost:3001/ | ||
|
|
||
| - Username: admin | ||
| - Password: admin | ||
|
|
||
| You should see the Temporal Cloud OpenMetrics dashboard. | ||
|
|
||
| ## 7) Verify metrics in Prometheus | ||
|
|
||
| Prometheus: http://localhost:9093/ | ||
|
|
||
| Go to: | ||
| Status → Targets (make sure the scrape target is UP) | ||
| Graph tab (search for Temporal metrics and run a query) | ||
|
|
||
| ## 8) Ran the sample and view the cloud metrics | ||
|
|
||
| - `./gradlew -q execute -PmainClass=io.temporal.samples.temporalcloudopenmetrics.WorkerMain` | ||
| - `./gradlew -q execute -PmainClass=io.temporal.samples.temporalcloudopenmetrics.Starter` |
54 changes: 54 additions & 0 deletions
54
core/src/main/java/io/temporal/samples/temporalcloudopenmetrics/Starter.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,54 @@ | ||
| package io.temporal.samples.temporalcloudopenmetrics; | ||
|
|
||
| import io.temporal.client.WorkflowClient; | ||
| import io.temporal.client.WorkflowOptions; | ||
| import io.temporal.client.WorkflowStub; | ||
| import io.temporal.samples.temporalcloudopenmetrics.workflows.ScenarioWorkflow; | ||
| import java.time.Duration; | ||
| import java.util.UUID; | ||
|
|
||
| public class Starter { | ||
|
|
||
| public static void main(String[] args) throws Exception { | ||
| WorkflowClient client = TemporalConnection.client(); | ||
|
|
||
| String name = "Temporal"; | ||
| String[] scenarios = {"success", "fail", "timeout", "continue", "cancel"}; | ||
|
|
||
| for (String scenario : scenarios) { | ||
| String wid = "scenario-" + scenario + "-" + UUID.randomUUID(); | ||
|
|
||
| WorkflowOptions.Builder optionsBuilder = | ||
| WorkflowOptions.newBuilder() | ||
| .setTaskQueue(TemporalConnection.TASK_QUEUE) | ||
| .setWorkflowId(wid); | ||
|
|
||
| // workflow timeout | ||
| if ("timeout".equalsIgnoreCase(scenario)) { | ||
| optionsBuilder.setWorkflowRunTimeout(Duration.ofSeconds(3)); | ||
| } | ||
|
|
||
| ScenarioWorkflow wf = client.newWorkflowStub(ScenarioWorkflow.class, optionsBuilder.build()); | ||
|
|
||
| System.out.println("\n=== Starting scenario: " + scenario + " workflowId=" + wid + " ==="); | ||
|
|
||
| try { | ||
| if ("cancel".equalsIgnoreCase(scenario)) { | ||
| WorkflowClient.start(wf::run, scenario, name); | ||
| Thread.sleep(2000); | ||
| WorkflowStub untyped = client.newUntypedWorkflowStub(wid); | ||
| untyped.cancel(); | ||
| untyped.getResult(String.class); | ||
| continue; | ||
| } | ||
|
|
||
| // normal synchronous execution | ||
| String result = wf.run(scenario, name); | ||
| System.out.println("Scenario=" + scenario + " Result=" + result); | ||
|
|
||
| } catch (Exception e) { | ||
| System.out.println("Scenario=" + scenario + " ended: " + e.getClass().getSimpleName() + " - " + e.getMessage()); | ||
| } | ||
| } | ||
| } | ||
| } |
74 changes: 74 additions & 0 deletions
74
core/src/main/java/io/temporal/samples/temporalcloudopenmetrics/TemporalConnection.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| package io.temporal.samples.temporalcloudopenmetrics; | ||
|
|
||
| import io.grpc.netty.shaded.io.grpc.netty.GrpcSslContexts; | ||
| import io.grpc.netty.shaded.io.netty.handler.ssl.SslContext; | ||
| import io.grpc.netty.shaded.io.netty.handler.ssl.SslContextBuilder; | ||
| import io.temporal.client.WorkflowClient; | ||
| import io.temporal.client.WorkflowClientOptions; | ||
| import io.temporal.serviceclient.WorkflowServiceStubs; | ||
| import io.temporal.serviceclient.WorkflowServiceStubsOptions; | ||
| import java.io.FileInputStream; | ||
| import java.io.InputStream; | ||
|
|
||
| public final class TemporalConnection { | ||
| private TemporalConnection() {} | ||
|
|
||
| // add your namespace here | ||
| public static final String NAMESPACE = env("TEMPORAL_NAMESPACE", "deepika-test-namespace.a2dd6"); | ||
|
|
||
| public static final String ADDRESS = | ||
| env("TEMPORAL_ADDRESS", "deepika-test-namespace.a2dd6.tmprl.cloud:7233"); | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧹 |
||
|
|
||
| public static final String CERT = | ||
| env("TEMPORAL_CERT", "/Users/deepikaawasthi/temporal/temporal-certs/client.pem"); | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧹 |
||
|
|
||
| public static final String KEY = | ||
| env("TEMPORAL_KEY", "/Users/deepikaawasthi/temporal/temporal-certs/client.key"); | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧹 |
||
|
|
||
| public static final String TASK_QUEUE = env("TASK_QUEUE", "openmetrics-task-queue"); | ||
|
|
||
| // default 60s; override with WORKER_SECONDS env var | ||
| public static final int WORKER_SECONDS = envInt("WORKER_SECONDS", 60); | ||
|
|
||
| public static WorkflowClient client() { | ||
| WorkflowServiceStubs service = serviceStubs(); | ||
| return WorkflowClient.newInstance( | ||
| service, WorkflowClientOptions.newBuilder().setNamespace(NAMESPACE).build()); | ||
| } | ||
|
|
||
| // this will create servicestubs and which will be used by both workermain and starter | ||
| private static WorkflowServiceStubs serviceStubs() { | ||
| try (InputStream clientCert = new FileInputStream(CERT); | ||
| InputStream clientKey = new FileInputStream(KEY)) { | ||
|
|
||
| SslContext sslContext = | ||
| GrpcSslContexts.configure(SslContextBuilder.forClient().keyManager(clientCert, clientKey)) | ||
| .build(); | ||
|
|
||
| WorkflowServiceStubsOptions options = | ||
| WorkflowServiceStubsOptions.newBuilder() | ||
| .setTarget(ADDRESS) | ||
| .setSslContext(sslContext) | ||
| .build(); | ||
|
|
||
| return WorkflowServiceStubs.newServiceStubs(options); | ||
| } catch (Exception e) { | ||
| throw new RuntimeException("Failed to create Temporal TLS connection", e); | ||
| } | ||
| } | ||
|
|
||
| private static String env(String key, String def) { | ||
| String v = System.getenv(key); | ||
| return (v == null || v.isBlank()) ? def : v; | ||
| } | ||
|
|
||
| private static int envInt(String key, int def) { | ||
| String v = System.getenv(key); | ||
| if (v == null || v.isBlank()) return def; | ||
| try { | ||
| return Integer.parseInt(v.trim()); | ||
| } catch (Exception e) { | ||
| return def; | ||
| } | ||
| } | ||
| } | ||
30 changes: 30 additions & 0 deletions
30
core/src/main/java/io/temporal/samples/temporalcloudopenmetrics/WorkerMain.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| package io.temporal.samples.temporalcloudopenmetrics; | ||
|
|
||
| import io.temporal.client.WorkflowClient; | ||
| import io.temporal.samples.temporalcloudopenmetrics.activities.ScenarioActivitiesImpl; | ||
| import io.temporal.samples.temporalcloudopenmetrics.workflows.ScenarioWorkflowImpl; | ||
| import io.temporal.worker.WorkerFactory; | ||
|
|
||
| public class WorkerMain { | ||
| public static void main(String[] args) throws Exception { | ||
| WorkflowClient client = TemporalConnection.client(); | ||
|
|
||
| WorkerFactory factory = WorkerFactory.newInstance(client); | ||
| io.temporal.worker.Worker worker = factory.newWorker(TemporalConnection.TASK_QUEUE); | ||
|
|
||
| worker.registerWorkflowImplementationTypes(ScenarioWorkflowImpl.class); | ||
| worker.registerActivitiesImplementations(new ScenarioActivitiesImpl()); | ||
|
|
||
| factory.start(); | ||
| System.out.println( | ||
| "Worker started. namespace=" | ||
| + TemporalConnection.NAMESPACE | ||
| + " taskQueue=" | ||
| + TemporalConnection.TASK_QUEUE); | ||
|
|
||
| Thread.sleep(TemporalConnection.WORKER_SECONDS * 1000L); | ||
|
|
||
| System.out.println("Stopping worker after " + TemporalConnection.WORKER_SECONDS + "s"); | ||
| factory.shutdown(); | ||
| } | ||
| } |
10 changes: 10 additions & 0 deletions
10
...main/java/io/temporal/samples/temporalcloudopenmetrics/activities/ScenarioActivities.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| package io.temporal.samples.temporalcloudopenmetrics.activities; | ||
|
|
||
| import io.temporal.activity.ActivityInterface; | ||
| import io.temporal.activity.ActivityMethod; | ||
|
|
||
| @ActivityInterface | ||
| public interface ScenarioActivities { | ||
| @ActivityMethod | ||
| String doWork(String name, String scenario); | ||
| } |
39 changes: 39 additions & 0 deletions
39
.../java/io/temporal/samples/temporalcloudopenmetrics/activities/ScenarioActivitiesImpl.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| package io.temporal.samples.temporalcloudopenmetrics.activities; | ||
|
|
||
| import io.temporal.activity.Activity; | ||
| import io.temporal.failure.ApplicationFailure; | ||
|
|
||
| public class ScenarioActivitiesImpl implements ScenarioActivities { | ||
|
|
||
| @Override | ||
| public String doWork(String name, String scenario) { | ||
| if ("fail".equalsIgnoreCase(scenario)) { | ||
| throw ApplicationFailure.newNonRetryableFailure( | ||
| "Intentional failure for dashboard", "SCENARIO_FAIL"); | ||
| } | ||
|
|
||
| if ("timeout".equalsIgnoreCase(scenario)) { | ||
| sleepMs(5_000); | ||
| return "unreachable (should have timed out)"; | ||
| } | ||
|
|
||
| if ("cancel".equalsIgnoreCase(scenario)) { | ||
| while (true) { | ||
| Activity.getExecutionContext().heartbeat("still-running"); | ||
| sleepMs(1_000); | ||
| } | ||
| } | ||
|
|
||
| // success | ||
| return "Hello " + name; | ||
| } | ||
|
|
||
| private static void sleepMs(long ms) { | ||
| try { | ||
| Thread.sleep(ms); | ||
| } catch (InterruptedException e) { | ||
| Thread.currentThread().interrupt(); | ||
| throw new RuntimeException("Interrupted", e); | ||
| } | ||
| } | ||
| } |
37 changes: 37 additions & 0 deletions
37
core/src/main/java/io/temporal/samples/temporalcloudopenmetrics/docker-compose.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| services: | ||
| grafana: | ||
| image: grafana/grafana:latest | ||
| ports: | ||
| - "3001:3000" | ||
| depends_on: | ||
| - prometheus | ||
| networks: | ||
| - temporal_network | ||
| volumes: | ||
| - ./grafana/provisioning:/etc/grafana/provisioning:ro | ||
| - ./grafana/dashboards:/var/lib/grafana/dashboards:ro | ||
|
|
||
|
|
||
| prometheus: | ||
| container_name: prometheus | ||
| image: prom/prometheus:v2.37.0 | ||
| command: | ||
| - --web.enable-remote-write-receiver | ||
| - --config.file=/etc/prometheus/prometheus.yml | ||
| ports: | ||
| - "9093:9090" # host port changed to avoid conflicts | ||
| volumes: | ||
| - type: bind | ||
| source: ./prometheus/config.yml | ||
| target: /etc/prometheus/prometheus.yml | ||
| - type: bind | ||
| source: ./secrets/temporal_cloud_api_key | ||
| target: /etc/prometheus/temporal_cloud_api_key | ||
| read_only: true | ||
| networks: | ||
| - temporal_network | ||
|
|
||
| networks: | ||
| temporal_network: | ||
| driver: bridge | ||
| name: temporal_network |
Binary file added
BIN
+490 KB
...src/main/java/io/temporal/samples/temporalcloudopenmetrics/docs/images/img1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+190 KB
...src/main/java/io/temporal/samples/temporalcloudopenmetrics/docs/images/img2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+193 KB
...src/main/java/io/temporal/samples/temporalcloudopenmetrics/docs/images/img3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🧹