Skip to content

Jikkou - kafka config #517

@rubinaprusty

Description

@rubinaprusty

Hi Team,

I am trying to run Jikkou within docker image - streamthoughts/jikkou-api-server:main in mac and it is running fine. Now I would like to automate kafka topic creation by passing a config yml file -
My kafka cluster is running locally using localhost:9092.
I am using below yml file to run the topic to be created in kafka cluster -

apiVersion: io.streamthoughts.jikkou/v1beta1
kind: ResourceList
resources:
  # Kafka connection configuration
  - apiVersion: jikkou.io/v1beta1
    kind: KafkaConfiguration
    metadata:
      name: kafka-cluster
    spec:
      bootstrap.servers: "localhost:9092"
      security.protocol: PLAINTEXT
      
  # Define a Kafka topic
  - apiVersion: kafka.jikkou.io/v1beta1
    kind: KafkaTopic
    metadata:
      name: test-topic-20012025
    spec:
      partitions: 3
      replicas: 2
      config:
        cleanup.policy: "delete"
        retention.ms: "604800000"  # 7 days
        max.message.bytes: "1048576"

My docker file -

# Use the base image for Jikkou API Server
FROM streamthoughts/jikkou-api-server:main
COPY kafka-topics.yml /config/kafka-topics.yml
# Default command to apply Kafka topic configurations using Jikkou
CMD ["jikkou", "apply", "--files", "/config/kafka-topics.yml"]

Now I am able to run Jikkou on my machine but could not able to connect to the kafka cluster.

below is my admin config -

AdminClientConfig values: 
        auto.include.jmx.reporter = true
        bootstrap.controllers = []
        bootstrap.servers = [localhost:9092]
        client.dns.lookup = use_all_dns_ips
        client.id = 
        connections.max.idle.ms = 300000
        default.api.timeout.ms = 60000
        enable.metrics.push = true
        metadata.max.age.ms = 300000
        metric.reporters = []
        metrics.num.samples = 2
        metrics.recording.level = INFO
        metrics.sample.window.ms = 30000
        receive.buffer.bytes = 65536
        reconnect.backoff.max.ms = 1000
        reconnect.backoff.ms = 50
        request.timeout.ms = 30000
        retries = 2147483647
        retry.backoff.max.ms = 1000
        retry.backoff.ms = 100
        sasl.client.callback.handler.class = null
        sasl.jaas.config = null
        sasl.kerberos.kinit.cmd = /usr/bin/kinit
        sasl.kerberos.min.time.before.relogin = 60000
        sasl.kerberos.service.name = null
        sasl.kerberos.ticket.renew.jitter = 0.05
        sasl.kerberos.ticket.renew.window.factor = 0.8
        sasl.login.callback.handler.class = null
        sasl.login.class = null
        sasl.login.connect.timeout.ms = null
        sasl.login.read.timeout.ms = null
        sasl.login.refresh.buffer.seconds = 300
        sasl.login.refresh.min.period.seconds = 60
        sasl.login.refresh.window.factor = 0.8
        sasl.login.refresh.window.jitter = 0.05
        sasl.login.retry.backoff.max.ms = 10000
        sasl.login.retry.backoff.ms = 100
        sasl.mechanism = GSSAPI
        sasl.oauthbearer.clock.skew.seconds = 30
        sasl.oauthbearer.expected.audience = null
        sasl.oauthbearer.expected.issuer = null
        sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
        sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
        sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
        sasl.oauthbearer.jwks.endpoint.url = null
        sasl.oauthbearer.scope.claim.name = scope
        sasl.oauthbearer.sub.claim.name = sub
        sasl.oauthbearer.token.endpoint.url = null
        security.protocol = PLAINTEXT
        security.providers = null
        send.buffer.bytes = 131072
        socket.connection.setup.timeout.max.ms = 30000
        socket.connection.setup.timeout.ms = 10000
        ssl.cipher.suites = null
        ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
        ssl.endpoint.identification.algorithm = https
        ssl.engine.factory.class = null
        ssl.key.password = null
        ssl.keymanager.algorithm = SunX509
        ssl.keystore.certificate.chain = null
        ssl.keystore.key = null
        ssl.keystore.location = null
        ssl.keystore.password = null
        ssl.keystore.type = JKS
        ssl.protocol = TLSv1.3
        ssl.provider = null
        ssl.secure.random.implementation = null
        ssl.trustmanager.algorithm = PKIX
        ssl.truststore.certificates = null
        ssl.truststore.location = null
        ssl.truststore.password = null
        ssl.truststore.type = JKS

10:27:02.791 [scheduled-executor-thread-1] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka version: 3.7.0
10:27:02.792 [scheduled-executor-thread-1] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka commitId: 2ae524ed625438c5
10:27:02.792 [scheduled-executor-thread-1] INFO  o.a.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1737455222791
10:27:02.794 [scheduled-executor-thread-1] INFO  i.s.j.k.internals.KafkaBrokersReady - Checking for Kafka to be ready. Expected broker(s): 1
10:27:32.799 [kafka-admin-client-thread | adminclient-52] INFO  o.a.k.c.a.i.AdminMetadataManager - [AdminClient clientId=adminclient-52] Metadata update failed
org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: fetchMetadata
10:28:02.800 [kafka-admin-client-thread | adminclient-52] INFO  o.a.k.c.a.i.AdminMetadataManager - [AdminClient clientId=adminclient-52] Metadata update failed

I am able to create the same topic using cli command -

Please help me here what I am doing wrong - I am guessing my configuration is wrong or provide me a sample config and docker file.

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions