How to avoid nodes to be quarantined in Akka Cluster?

Akka Libraries Akka Cluster
,
1

The doc of Akka Remoting (Artery) describes quarantine as follows:

a1
a1.png836×232 26.7 KB

Nodes will be quarantined when remote failure detector triggers in Akka Remote, but will not in Akka Cluster.

The doc of Akka Cluster describes quarantine as follows:

a2
a2.png852×125 17.2 KB

Nodes will be quarantined when there are too many unacknowledged system messages, but will not be quarantined when failure detector triggers.

So I have 3 questions:

  1. In Akka Cluster, failure detector triggers don’t cause nodes to be quarantined, so what are their roles?
  2. “too many unacknowledged system messages”, what’s the number of “too many” ? Can we addjust it ?
  3. How to config in Akka Cluster to make nodes will not or be harder to be quarantined ?

Thanks in advance.

2

To detect network problems and crashed nodes. If the heartbeat messages (requrest-reply) can’t get through (lost or delayed) it will mark them as Unreachable. When heartbeats can get through again it will be marked as reachable again. This doesn’t mean that nodes are removed from the cluster membership.

To decide when an Unreachable node should be removed from the cluster it has to be Downed. That is done by a downing provider, such as Lightbend’s Split Brain Resolver, or manually with Cluster management tool.

Some cluster tools, such as Cluster aware routers, use the reachability information to avoid routing messages to unreachable nodes.

That should be rare, but for example if many actors stop at the same time and there are watchers of these actors on other nodes there may be a storm of Terminated messages sent more quickly than they can be delivered and thereby filling up buffers.

The default size of the system messages bufffer is 20000 and it can be increased with configuration property akka.remote.artery.advanced.system-message-buffer-size. There is no drawback apart from possible memory consumption to increase this. The buffer is an ArrayDeque so it grows as needed, but doesn’t shrink.

There is also another queue for outgoing control (system) messages and the max size of that is configured with akka.remote.artery.advanced.outbound-control-queue-size. The default is 3072. This is a LinkedBlockingQueue so it’s also ok to increase. I think we should increase the default of this, by the way.

Answer to previous question covers this as well.

1 Like
3

Great answer! Thanks very much!