Hi,
I want to run a stream in an actor and make that fault tolerant on a cluster. What are the best practices for that?
Akka sharding sounds promising but then the automatic rebalancing is an issue as this would interrupt my stream.
At this point, I have a hand crafted master / worker approach. Where I create workers with a ClusterRouterPool / AdaptiveLoadBalancingPool as the idea was to pick a node in the cluster with low load. When one of my stream nodes is terminated I increase the pool size and assign the new worker the stream that failed. However, I run into a big https://github.com/akka/akka/issues/25385 and learned that AdaptiveLoadBalancingPool is changing the size of the pool dynamically what is not documented.
Sorry, my bad stating that, the AdaptiveLoadBalancing doesn’t do any dynamic resizing, except for when you remove and add nodes to the cluster.
If automatic rebalance would be problematic with your stream, what do you do when a new node is added to your cluster, or you need to remove the node a stream is running on, or when a node have crashed?
Additionally choosing the right node™ to run a stream on is trickier than choosing the right node to run a small:ish workload on, since a stream is in general long lived, imagine five requests coming at roughly the same time, all seeing node N having low load and starting long lived streams on that node.
Best might be to make sure you have a way to shard requests fairly on some identifier/property of the request.
I think I mentioned in the gitter discussion a couple of days ago, but you will also have to think about back pressure and not overwhelming the worker nodes. Whatever solution you choose, make sure you try a higher inflow of work than you can complete out and see in what way that affects the solution. (This is usually why you should avoid routers for distributing workloads)
Really good point - thank you. Let me think about this a little more. I also plan to do some code reading of GearPump that you mentioned in your talk to understand how they handled these kinds of scenarios.
Do you happen to know if there is anyway to switch off / overwrite the Akka sharding redistribution?
You can write your own ShardAllocationStrategy to make your own decisions about where to allocate new shards and when to rebalance.
I’m not sure there is much happening in Gearpump land, looking at their GitHub repo the last change was two years ago, but maybe they are developing it elsewhere. I don’t think I have mentioned it in talks for quite a while so maybe look for a newer talk if you are using one as reference. :)
Thanks @johanandren
Yes - the talk you mentioned GearPump is a little older.
I’m still trying to solve my problem described above. Now that using a ClusterPool is not possible I’m trying to deploy remote actors manually and than create a group from them. However, I can’t find an example how to create an Actor manually on a different cluster node. Do you happen to have a snippet or pointer?
Thanks.
S
I’d recommend that you try to turn the problem around and have nodes create their own actors and then have them take part in coordinated work rather than using central management of remote deployed actors (in addition to making the distributed app more tightly coupled that also introduces a single point of failure as the all remotely deployed actors will have a parent on the node that spawned them).
If you look at pretty much all the cluster tool you will notice that this is how they are designed.
