Downing an unreachable node via akka management doesn't solve the unreachability

Akka Libraries Akka Cluster
1

Hello, we have a cluster of 5 nodes and we use akka 2.8.6. We do not use the auto-downing feature but instead we down unreachable nodes via akka management.
Today a node in our cluster became unreachable to all other 4 nodes in the cluster. we used the following down operation of akka management:

curl -X PUT -d operation=Down http://<IP>:<PORT>/cluster/members/akka://actorSystem@<UNREACHABLE_NODE_IP>:<UNREACHABLE_NODE_PORT>'

We noticed that the state of the downed node became “Down” for all other nodes in the cluster, however it remained Unreachable to all of them. What can be the reason of this state and how to solve the unreachability at this point?

2

Have you tried leave operation after the Down?

3

Yes but it does not work. The node has crashed and it needed to be removed from the cluster forcibly. Asking for a Graceful shutdown using the Leave operation doesn’t work as the node is unable to remove itself from the cluster.