Hello, we are using Kubernetes for Akka clustering deployment and management.
Basically, we have 3 replicas or pod to build Akka clustering.
Each pod application code is subscribing Kafka topic with parallel = 3
Like
service.applicationTopic
.subscribe.atLeastOnce(Flow[ApplicationMsg]
.mapAsync(parallelism = 3) { ... }
As far as I understand, the sample code can make each application pod have up to 3 consumer connection pools. And we have 3 pods so the cluster will have up to 9 connection pools for the topic. (Btw number of Kafka partition is 8)
We recently decided to increase the number of pod for scale out for better performance.
If we increase the number of pod from 3 to 7, and each application can have parallel = 3 thread pool will try to connect to Kafka broker of 8 partitions.
In this case, is there any possibility that some pods that deployed earlier than others could occupy all the Kafka broker connections? And some late pods don’t have chance to subscribe the topic due to the limit of Kafka partitions?
And should I decrease the parallel value so that I can expect each pods can have fair chance to connect to each Kafka partitions?