Naive question, but for what reason would you prefer to down
a node (manually), opposed to making a node leave
? What is the utility of downing
if sharding and singleton migration does not occur, as I understand the docs?
Leaving is the graceful variant while downing is what you do for a node that is no longer in the cluster, for example because of it crashing (and becoming unreachable). So in short, if you want to remove a healthy node leave
(it will also do this on its own when triggering the coordinated shutdown for example because of the process getting killed with sigterm), if you want to remove a node that has crashed, use down
.
Both will trigger shard and singletons to run on other nodes after the node has been removed, but in the case of downing the migration cannot be a graceful handoff because it may not be possible to communicate with the node that is being downed. In case of a transient network problem it could hovewer become reachable just after the decision to down it was taken, and receive a message telling it about this, and then that will, by default, trigger coordinated shutdown but not the handover logic.
Some more related details around network partitions and dealing with unreachable nodes can be found in the Split Brain Resolver docs here.
Great, thanks, that clarifies things a lot.
So the big difference is that leaving can only be performed while the cluster is in a consistent state, while downing can be performed at any point. Downing can be used to restore the cluster to a consistent state.
As an experiment, I tried to leave
a node that was unreachable. The node just remains in a leaving
state, unless you then mark the node as down
.
Edit: And to add to that, importantly, shards and singletons will be allocated to another node