In our use case, we have hundreds of streams consuming from hundreds of Kafka topics (thousands of partitions in total) and writing that data to just one topic. In the downstream topic (where our streams are producing the data) we see a huge number of TCP connections getting created. We did some analysis and found out that the number of connections is equal to the number of partitions the stream is reading from (here we have thousands of partitions as explained above).
We are using alpakka kafka 3.0.1 jar. We are using Producer.flexiFlow to produce the messages.
Have you seen that you can share a single underlying Kafka producer between multiple streams producing to Kafka as described here in the docs?
The documentation illustrates sharing a Kafka Producer in the sink phase. We are using kafka producer as a flow not as sink.
Passing a ProducerSettings created with a shared producer to the flow factory should give the same as for the Sinks, leading to all producing to Kafka going through that producer.