Yes, it is possible to know whether the system is terminated as expected or crashed due to an unexpected error. In Akka, the Coordinated Shutdown process is used to manage the graceful shutdown of an application. This process is triggered when a node is sent a SIGTERM signal, not a SIGKILL.
During the Coordinated Shutdown, the system will perform a Cluster(system).leave on itself, change the status of the member to Exiting, allow any shards to be shutdown gracefully, and migrate ClusterSingletons if this was the oldest node. Finally, the node is removed from the Akka Cluster membership. This process ensures that the system is terminated as expected.
However, if an error occurs, such as a crash, the system may be terminated unexpectedly. In such cases, the system may not have time to run the Coordinated Shutdown process. For example, if the guardian actor crashes on start, the system is immediately terminated, and there is no actor system to keep running the coordinated shutdown anymore.
In general, the Coordinated Shutdown will be most important for things that need some graceful shutting down, such as sharding waiting for stopping entities, Akka HTTP/gRPC giving in-flight requests time to complete before closing connections, and cluster leaving instead of relying on SBR to kick a node out.
To handle such scenarios, Akka provides the Split Brain Resolver module. This module has a number of strategies that can ensure that the cluster continues to function during network partitions and node failures.
Sources: