Add jaeger tracing

[ethanfrey]

Follow up from #100

We would also like to allow configuring the tracing to dump to other formats than stdout.

One of the most interesting outputs is jaeger (or other opentracing server), which involves adding spans in places. This also means tracking spans not just in one place in the pipeline, but following the action from trigger pickup to dispatcher to wasm engine to submission of the result on the chain.

This will probably be a lot more work than nice text logging, so pulled into another milestone to give time to do it right. You can look at https://github.com/lay3rlabs/layer-sdk for another example of doing this with jaeger

---

Layer SDK docker references:

• You can use this image (or a newer version) in docker-compose to handle everything. This is good for local: https://github.com/Lay3rLabs/layer-sdk/blob/main/localnode/docker-compose.yml#L59-L67
• If you want something for long-lived testnet (several million events) with persistence and memory limits, you can look at https://github.com/Lay3rLabs/layer-sdk/blob/main/localnode/jaeger-elastic-compose.yml but that should be out of scope of this issue
• We pass in a config line to redirect to the jaeger host for dumping events (mind the port numbers, they can be tricky here) https://github.com/Lay3rLabs/layer-sdk/blob/main/localnode/abci/config/slay3r.toml#L2

Layer SDK code references:

The tracing setup code is here: https://github.com/Lay3rLabs/layer-sdk/blob/main/app/slay3rd/src/main.rs#L78-L109 please not the following points:

• It uses config.filter to filter events. Which is basically this filter from WAVS - important to keep that
• Current WAVS setup only uses stdout to log tracing, you probably want to support both options - either stdout or jaeger like the layer-sdk setup. You should compare the two approaches to building the fmt subscriber and define what works best for you.
• You can try copying the tracing-jaeger setup from layer-sdk, but it is deprecated and you are supposed to use otlp-tracer now instead. I did try this with layer-sdk some months ago, but I couldn't get it to work. Check out my PR to see my (lack of) progress, which may be a good learning experience, not a template.
  • One idea I had was a minimal test case to see if this works at all, and then when you get a Rust binary writing to the Jaeger docker image, you can then port it over to WAVS with a lot more confidence and a quicker testing cycle. This article gives such a sample app. I'd try this out and connect to jaeger docker (and expose some ports as needed) and then take those learnings to implement this PR properly

[dakom]

Bumping this up, it's crucial for DevEx to be able to debug components.

In terms of implementation, opentelemetry-jaeger has been deprecated since LayerSDK, and it's a bit confusing to figure out which packages are the current state of the art. Here's ChatGPT's suggestion as of now (untested):

Cargo.toml

[dependencies]
tracing = "0.1"
tracing-subscriber = "0.3"
tracing-opentelemetry = "0.19"
opentelemetry = { version = "0.18", features = ["rt-tokio"] }
opentelemetry-otlp = "0.10"

WAVS code:

use opentelemetry::runtime::Tokio;
use opentelemetry_otlp::WithExportConfig;
use tracing_subscriber::{layer::SubscriberExt, Registry};

fn init_tracer() -> Result<(), Box<dyn std::error::Error>> {
    // Build an OTLP pipeline that exports spans to an OTLP endpoint.
    let tracer = opentelemetry_otlp::new_pipeline()
        // Point to your collector’s OTLP endpoint (gRPC, by default, listens on 4317)
        .with_endpoint("http://localhost:4317")
        .install_batch(Tokio)?;

    // Create a tracing layer that sends spans to OpenTelemetry.
    let otel_layer = tracing_opentelemetry::layer().with_tracer(tracer);

    // Set up a subscriber that combines the OpenTelemetry layer and a human-readable log layer.
    let subscriber = Registry::default()
        .with(otel_layer)
        .with(tracing_subscriber::fmt::layer());

    tracing::subscriber::set_global_default(subscriber)?;
    Ok(())
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    init_tracer()?;

    tracing::info!("Application started!");

    // Your application logic goes here...
    
    // Flush any remaining spans before shutdown.
    opentelemetry::global::shutdown_tracer_provider();
    Ok(())
}

(note: opentelemetry-jaeger-propagator is available if we need to mix with other services... I'm not sure exactly how it works, but iiuc it's more relevant when we want to mix in e.g. aggregator at the same time, but I think that can be a separate issue and we should keep the first implementation without that)

Docker compose

version: '3'
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC receiver
  jaeger:
    image: jaegertracing/all-in-one:1.35
    ports:
      - "16686:16686"  # Jaeger UI

otel-collector-config.yaml configuration

receivers:
  otlp:
    protocols:
      grpc:

exporters:
  jaeger:
    # For the Jaeger exporter, specify the Jaeger collector endpoint.
    endpoint: "jaeger:14250"
    tls:
      insecure: true

processors:
  batch:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]

Usage

1. Start the Collector & Jaeger with docker-compose up

2. Run WAVS

3. Access Jaeger UI at http://localhost:16686

[ethanfrey]

I tried migrating from opentelemetry-jaeger and moving to the new form, but then had trouble getting anything feeding into jaeger properly. I spent a full day with this in September (or October) on layer-sdk but it didn't work and I got no useful debug statements.

In retrospect, probably easiest to try this out in a dummy scenario, not with 100s of moving parts and tokio collectors and batching and async issues. And once that is done, try to integrate it with the product

That said, I can't give help here, just some advise on how to approach it. If you are stuck, let me know and I can try to help

[ethanfrey]

I found this one: https://github.com/open-telemetry/opentelemetry-rust/blob/main/examples/tracing-jaeger/src/main.rs

which does use tonic, but notably does not set the host/port (should be a default in the exporter that can be overriden).

Maybe you can get that working in a simple dockerfile, also to ensure you have the proper jaeger docker image that supports that and it is all wired up, then drop this jaeger in and connect with the wavs binary