Set Kafka Topic Configuration

Lagom Lagom Persistence API
,
1

Currently, after I let lagom create my kafka topic (as per my service description), I have to run

bash $kafka_installation/bin/kafka-configs.sh \
    --zookeeper $zookeeper_server:2181 \
    --alter \
    --entity-type topics \
    --entity-name $topic \
    --alter \
    --add-config retention.ms=3600000

Is there a way I can configure lagom to set this automatically upon creation of the topic so that I dont have to manually run my script every time a topic is created?

Thanks,
Franz

2

Lagom does not create topics itself. Most likely, you have the auto.create.topics.enable property turned on in your Kafka broker configuration. This will use the broker’s default retention policy for automatically-created topics. You can update the broker’s default setting, or create your topics explicitly instead of relying on auto-creation (this is recommended).

1 Like
3

Hi @TimMoore

Thanks for the reply.

We did not set auto.create.topics.enable so I guess the default is true for that. But if “lagom does not create topics itself”, and auto.create.topics.enable is set to true, which library does create the topics?

Thanks

4

That’s correct, it’s true by default. This is a broker configuration, so the topic is created by the broker itself whenever a client tries to publish a message to a topic that doesn’t already exist. It is typically recommended to disable this in production.

See the Kafka documentation for more details http://kafka.apache.org/documentation/

5

@TimMoore I see. So what you’re saying is that auto.create.topics.enable is a kafka broker configuration (not in lagom). So when lagom publishes an event to kafka, it doesnt actually check whether the topic exists or not. And because of auto.create.topics.enable, once kafka receives an event for a topic that does not exist, it will auto-create that topic.

Did I get that right?

6

Yes, exactly right! Sorry that I wasn’t more clear in the first place.

7

Thanks @TimMoore. No, you were clear :) I just want to reiterate to verify my understanding :smiley:

Thanks! I’ll just try and modify our microservice to create event topics via kafka.admin.AdminUtils#createTopic()

Thanks!