Hi Team
Does each message/command send to a sharded actor has got to be forwarded all the way thru Shard region coordinator->Shard Actor-> Entity(the intended recepient) ?
If thats the case the Shard region co-ordinater might get congested with a all of the messages sent to all of the actors under all the shards under that coordinator in the node right?
Could some one please explain how this could be overcome?
Thank you
Anand Krishnan
Hi @iamanandkris,
That’s not exactly how it works. The ShardCoordinator is not a hub where everything has to go through.
Instead, this is what will happen:
Imagine a 3-node (A, B, C) cluster each hosting 2 shard regions (A1, A2, B1, B2, C1, C2).
When node C gets a message that is supposed to go to shard B2, it will first ask the ShardCoordinator where it can find B2. It won’t send the message to the coordinator to get it dispatched to the right location. If will ask for the location instead.
After that first call, subsequent messages arriving in C and targeting B2 won’t need a round trip to the ShardCoordinator, because C now knows where B2 lives. After a while, all nodes will know where are all the shards.
Now, that was the situation in the past. Since the introduction of ddata, the shard information is automatically shared between the nodes using a CRDT. In that case, when C become part of a cluster it almost immediately gets to know the shard locations.
And when we have a shard re-distribution, for instance, when a node is removed and its shard regions are moved to other nodes, that information is also propagated to all remaining nodes via the CRDTs.
UPDATE: as @patriknw mentioned below, the ShardCoordinator still need to be contacted whenever the location of an entity is unknown, also when using ddata.
In which case the message is still sent via the shard actor for all the entities in that shard? i.e. it is not possible to send directly to the entities? In reality (normal use cases) I can’t see that being a bottleneck as no processing in the shard, just forward. did I understand this correctly?
Or using the entityIdExtractor logic, can the ShardCoordinator in C resolve the address of the Entity Actor residing B2?
Or has it got to be from - SenderActor -> C’s ShardCoordinator -> B2 Region -> Recipient ?, because at the time when C's ShardCoordinator receives a message, it does not know the Recipient Actor in B2Region is existing or not right? So B2 Region has to make sure that Recipient is created
I think that there are a couple of concepts being mixed here.
The ShardCoordinator is not dispatching messages to final recipients. His role is to decide how the shards are distributed. There is only one ShardCoordinator (cluster singleton), not one per node or shard.
However, you are right about the forwarding part. A message is delivered to a ShardRegion, then to a Shard and then to the Entity (final recipient). There is no processing and message are just forwarded.
As you also mentioned, the ShardRegion must make sure that Entities under its responsibility get initialized when needed. Basically this kind of knowledge belongs to the ShardRegion and is managed by it while the central coordinator make sure that each node gets its share and keep tracking of the locations.
You may be interested in reading this part of the docs.
https://doc.akka.io/docs/akka/2.5/cluster-sharding.html?language=scala#how-it-works
I hope this can help clarify it.
Cheers
To clarify, a shard is a group of entities (actors) and it’s only for the first message from a node to any of the entities in a shard that the coordinator is involved. After that the location of the shard is cached on the sending node. E.g. a shard may consist of 10,000 entities.
Those caches are invalidated when shards are moved, rebalance or crash, but in general the coordinator is not used involved in ”normal” message delivery.
BTW, this didn’t change with ddata. The regions still ask the coordinator (single source of truth) for the location of shards.
Thanks for clarifying this. My assumption was not correct in the sense that I considered that the round trip to the coordinator was not needed when using ddata. Indeed the coordinator being the source of truth one must refer to it when looking up for location.
Hi,
We are working on a POC based on Akka Cluster 2.6.4 (Not using classic). I’m basically working on the idea of a stateful microservice to meet the SLA of 2-3 ms at max. What I found was that I’m able to mee this SLA for a READ request if the entity is on the same node where the request first lands for a Shard Region. Eg. HTTP Client -> Node 1(SR1) -> Entity 1. But if scenario is Http Client -> Node1(SR1) -> Node2 (SR2) -> Entity 2 then the performance dips to 5-6 ms. This seems to be a genuine scenario as data is getting transferred between JVM(s). But is there a way to optimize this?
Regards,
Tarun
Hi Tarun,
Unfortunately, there is no way to have the kind of stateful application that spans across multiple nodes without incurring the latency cost. You’ll have to make sure the requests end at the correct node one way or another, and all ways involve some type of coordination.
If I were in your shoes, I would try to argue the fact that the speed of light ain’t changing anytime soon, to quote Pat Helland.
Borislav
It’s right there will typically be a network hop to route the message to the node where the entity is hosted. On the other hand, if the entity has been used recently it can can serve the request immediately without loading state from a backend database. Saving a different network hop and database contention.
Additional latency of 3 ms for this roundtrip sounds a bit hight to me but depends on your environment. Serialization can also have a big impact, but I assume that you don’t use Java serialization.