I am building an application where I have 100+ streams, one stream corresponding to each kafka topic. I would like to control each stream independently, so individual streams are better option. If KafkaConsumer is not shared, each stream creates a new Consumer object which internally creates one heartbeat thread per stream. This means that the application would have 100+ heartbeat threads.
I tried using the Consumer.committableExternalSource where we can share the kafkaConsumerActor, but that causes a separate set of challenges. With this, the subscription becomes manual, so offset tracking and assignment becomes the application’s responsibility. Also, this api doesn’t support rebalancing which again becomes the application’s responsibility.
Is there any reason why KafkaConsumerActor is not sharable in Consumer.committableSource? Is there a workaround to share the kafkaConsumerActor in committableSource? I looked at the alpakka-kafka source, but I couldn’t find a way to override this behaviour.