(1) Is there a setting or any way to control the frequency with which the TopicProducer thread polls for events in the database to publish to Kafka? There is an equivalent setting (eventual consistency) for the read side tables for Cassandra.
(2) The Lagom documentation points to 3 different config files (Lagom Kafka client, Akka stream kafka, and Apache Java Kafka client) that have settings that govern the behavior of Lagom’s Kafka Broker.
https://www.lagomframework.com/documentation/1.4.x/scala/KafkaClient.html
- Do either of the three config files have a setting for item 1 above?
- Do batch.size and linger.ms in the Producer Configs affect item 1 - speed at which events can be published to Kafka?
We are noticing delays of ~2-3 seconds between events being published to Kafka from our Lagom service and exploring all possible ways to bring reduce this delay and control it better (e.g. tradeoff with server load).