DistributedPubSubMediator - Horizontal scalability?

[stephen-lane]

Hi Folks,

Thanks in advance for reading my topic,

Im looking at the akka DistributedPubSubMediator, and it would be perfect for my use case, except i’m not sure about its horizontal scalability. In the doco, it says that the pub/sub registry is shared via the gossip protocol between all akka nodes. In a scenario with many small pub/sub channels (i.e one or two subscribers for each topic) this doesn’t seem scalable.

If there are millions of pub/sub channels with millions of subscribers across across several nodes, will it reach a point where adding extra nodes will not add extra capacity due to having to replicate the same pub/sub registry across all nodes?

[patriknw (Patrik Nordwall)]

It’s not intended for millions of topics and would not work well for frequently created short lived topics.

@stephen-lane

If there are millions of pub/sub channels with millions of subscribers across across several nodes, will it reach a point where adding extra nodes will not add extra capacity due to having to replicate the same pub/sub registry across all nodes?

Yes, that is correct.

You need to create some intermediate level of less and stable number of topics running on each node and then perform more fine grained filtering to final destinations locally on each node.

[stephen-lane]

Excellent, thank you for the explanation, I hadn’t previously though about the solution of having a course grained topic for pub/sub, then filtering down within the local node.