outboundControlIngress not initialized yet

Akka Libraries Akka Cluster
1

Hi

I have a cluster of two nodes in two different subnets. At first I bring up the seed node in one subnet which comes up fine. Then I startup the other node in the another subnet, at that point I see the below error in the seed node. And the log says handshake with the other node did not complete in 2000ms. The log entries on the also says the handshake with the seed node didn’t complete and had a HandshakeTimeoutException.

Please respond if you have a clue what is wrong here. thanks !

Error in seed node:

2018-06-08 14:41:37.809 [proteus-stage-akka.actor.default-dispatcher-11] akka.actor.RepointableActorRef
ERROR: Error in stage [akka.remote.artery.InboundHandshake@6dd024da]: outboundControlIngress not initialized yet

java.lang.IllegalStateException: outboundControlIngress not initialized yet
	at akka.remote.artery.Association.outboundControlIngress(Association.scala:226)
	at akka.remote.artery.Association.sendControl(Association.scala:313)
	at akka.remote.artery.ArteryTransport.sendControl(ArteryTransport.scala:697)
	at akka.remote.artery.InboundHandshake$$anon$2.$anonfun$onHandshakeReq$1(Handshake.scala:207)
	at akka.remote.artery.InboundHandshake$$anon$2.akka$remote$artery$InboundHandshake$$anon$$after(Handshake.scala:229)
	at akka.remote.artery.InboundHandshake$$anon$2.akka$remote$artery$InboundHandshake$$anon$$onHandshakeReq(Handshake.scala:206)
	at akka.remote.artery.InboundHandshake$$anon$2$$anon$3.onPush(Handshake.scala:182)
	at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:519)
	at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:411)
	at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:588)
	at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:472)
	at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:563)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:745)
	at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:760)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:670)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2018-06-08 14:41:38.378 [proteus-stage-akka.actor.default-dispatcher-11] akka.actor.RepointableActorRef
ERROR: Error in stage [akka.remote.artery.InboundHandshake@72fc1d13]: outboundControlIngress not initialized yet
2

That is something we would like to look into and fix if it’s a bug. Do you use latest version, 2.5.13? Please create an issue with description of how to reproduce.

3

Not sure what it is. I am not seeing the same issue when switching to ‘tcp’.
I was also seeing repeated PhiAccrualFailureDetector hearbeat interval warnings. Even though the nodes were ideal and there appeared to be no reason for delay in heartbeat exchange.
It was occurring even after increasing the failure-detector.threshold to 12 and failure-detector.acceptable-heartbeat-pause to 10s, where as my network card is 10000Mbs.
I am not seeing any of these after switching to tcp.
My machine has multiple network interfaces btw.

4

Sounds your network setup doesn’t work for UDP. I don’t know what could be wrong. Firewall settings?