gracefulStop in cluster mode - "cannot enqueue after timer shutdown"

Akka Libraries Akka Cluster
1

Hi ! Im seeing the following exception

java.lang.IllegalStateException: cannot enqueue after timer shutdown
	at akka.actor.LightArrayRevolverScheduler.scheduleOnce(LightArrayRevolverScheduler.scala:136)
	at akka.actor.Scheduler.scheduleOnce(Scheduler.scala:205)
	at akka.actor.Scheduler.scheduleOnce$(Scheduler.scala:202)
	at akka.actor.LightArrayRevolverScheduler.scheduleOnce(LightArrayRevolverScheduler.scala:37)
	at akka.pattern.PromiseActorRef$.apply(AskSupport.scala:604)
	at akka.pattern.GracefulStopSupport.gracefulStop(GracefulStopSupport.scala:48)
	at akka.pattern.GracefulStopSupport.gracefulStop$(GracefulStopSupport.scala:46)
	at akka.pattern.package$.gracefulStop(package.scala:42)
	at akka.pattern.Patterns$.gracefulStop(Patterns.scala:238)
	at akka.pattern.Patterns.gracefulStop(Patterns.scala)

when performing gracefulStop on an actor in cluster mode:

Cluster cluster = Cluster.get(myActorSystem);
cluster.leave(cluster.selfAddress());
Thread.sleep(1000);
Patterns.gracefulStop(actorRef, Duration.create(1, TimeUnit.MINUTES))

Logs say

11:39:30 [INFO ] [a.c.Cluster(akka://my)                  ] [my-akka.actor.default-dispatcher-3] - Cluster Node [akka.tcp://my@127.0.0.1:2552] - Marked address [akka.tcp://my@127.0.0.1:2552] as [Leaving]
11:39:31 [INFO ] [a.c.Cluster(akka://my)                  ] [my-akka.actor.default-dispatcher-2] - Cluster Node [akka.tcp://my@127.0.0.1:2552] - Exiting (leader), starting coordinated shutdown
11:39:31 [INFO ] [a.c.Cluster(akka://my)                  ] [my-akka.actor.default-dispatcher-1] - Cluster Node [akka.tcp://my@127.0.0.1:2552] - Leader is moving node [akka.tcp://my@127.0.0.1:2552] to [Exiting]
11:39:31 [INFO ] [a.c.Cluster(akka://my)                  ] [my-akka.actor.default-dispatcher-2] - Cluster Node [akka.tcp://my@127.0.0.1:2552] - Exiting completed
11:39:31 [INFO ] [a.c.Cluster(akka://my)                  ] [my-akka.actor.default-dispatcher-2] - Cluster Node [akka.tcp://my@127.0.0.1:2552] - Shutting down...
11:39:31 [INFO ] [a.c.Cluster(akka://my)                  ] [my-akka.actor.default-dispatcher-2] - Cluster Node [akka.tcp://my@127.0.0.1:2552] - Successfully shut down
11:39:31 [INFO ] [emoteActorRefProvider$RemotingTerminator] [my-akka.actor.default-dispatcher-3] - Shutting down remote daemon.
11:39:31 [INFO ] [emoteActorRefProvider$RemotingTerminator] [my-akka.actor.default-dispatcher-3] - Remote daemon shut down; proceeding with flushing remote transports.
11:39:31 [INFO ] [a.r.Remoting                            ] [my-akka.actor.default-dispatcher-3] - Remoting shut down
11:39:31 [INFO ] [emoteActorRefProvider$RemotingTerminator] [my-akka.actor.default-dispatcher-2] - Remoting shut down.
11:39:31 [ERROR] [m.ShutdownOperation                     ] [qtp46494202-112] - Unable to execute operation class my.ShutdownOperation
java.lang.IllegalStateException: cannot enqueue after timer shutdown
	at akka.actor.LightArrayRevolverScheduler.scheduleOnce(LightArrayRevolverScheduler.scala:136)

Is this expected ? Akka 2.5.12. Without the “Thread.sleep(1000);” the gracefulStop seems to do ok (most of the time it seems).

2

The reason is that the actor system has been shutdown when you do the gracefulStop call. The actor is also stopped when the ActorSystem is shutdown.

3

@patriknw indeed, thx for your response.

What is the proper way to perform node graceful stop in cluster setup ?
Do i need to look into the “Coordinated Shutdown” extension or is there a simpler way ?

My naive solution was

  • leave cluster (=> other nodes stop sending me messages)
  • do graceful stop on actors (=> digest existing message queues)
  • shutdown actor system
    which is not working for the reasons you described.
4

Coordinated shutdown is the way to do it. You can register own tasks in the phases, for example stopping actors if they require some special graceful handling.

Coordinated shutdown is integrated with cluster leaving and will also shutdown the ActorSystem as the last task.

5

Thank you @patriknw for the useful hints, i really appreciate it. I will dive into coordinated shutdown. Thank you !