Solved: Cluster Client gets dead letter messages on creation

Running on my laptop I have a single node cluster on localhost and I have a cluster client on the same host.

Here is the logs tail from the cluster (ClusterClientReceptionist listener is active):

10:04:11.463 TKD [ICEClusterSystem-akka.actor.default-dispatcher-3] INFO  akka.remote.Remoting - Starting remoting
10:04:11.950 TKD [ICEClusterSystem-akka.actor.default-dispatcher-3] INFO  akka.remote.Remoting - Remoting started; listening on addresses :[akka.tcp://ICEClusterSystem@127.0.0.1:2552]
10:04:11.975 TKD [ICEClusterSystem-akka.actor.default-dispatcher-3] INFO  akka.remote.Remoting - Remoting now listens on addresses: [akka.tcp://ICEClusterSystem@127.0.0.1:2552]
10:04:12.070 TKD [ICEClusterSystem-akka.actor.default-dispatcher-3] INFO  a.c.Cluster(akka://ICEClusterSystem) - Cluster Node [akka.tcp://ICEClusterSystem@127.0.0.1:2552] - Starting up...
10:04:12.289 TKD [ICEClusterSystem-akka.actor.default-dispatcher-3] INFO  a.c.Cluster(akka://ICEClusterSystem) - Cluster Node [akka.tcp://ICEClusterSystem@127.0.0.1:2552] - Registered cluster JMX MBean [akka:type=Cluster]
10:04:12.289 TKD [ICEClusterSystem-akka.actor.default-dispatcher-3] INFO  a.c.Cluster(akka://ICEClusterSystem) - Cluster Node [akka.tcp://ICEClusterSystem@127.0.0.1:2552] - Started up successfully
10:04:12.700 TKD [ICEClusterSystem-akka.actor.default-dispatcher-20] INFO  a.c.Cluster(akka://ICEClusterSystem) - Cluster Node [akka.tcp://ICEClusterSystem@127.0.0.1:2552] - Node [akka.tcp://ICEClusterSystem@127.0.0.1:2552] is JOINING, roles [backend, dc-default]
10:04:12.751 TKD [ICEClusterSystem-akka.actor.default-dispatcher-20] INFO  a.c.Cluster(akka://ICEClusterSystem) - Cluster Node [akka.tcp://ICEClusterSystem@127.0.0.1:2552] - Leader is moving node [akka.tcp://ICEClusterSystem@127.0.0.1:2552] to [Up]
10:04:14.230 TKD [ICEClusterSystem-akka.actor.default-dispatcher-17] INFO  a.c.Cluster(akka://ICEClusterSystem) - Cluster Node [akka.tcp://ICEClusterSystem@127.0.0.1:2552] - Metrics collection has started successfully
10:04:14.233 TKD [run-main-0] INFO  IceCluster - Registering iceClient(akka://ICEClusterSystem/user/iceClient) with ClusterClientReceptionist; akka://ICEClusterSystem/system/receptionist
10:04:14.316 TKD [ICEClusterSystem-akka.actor.default-dispatcher-17] INFO  IceClusterReceptionistListener - ClusterReceptionist: New set of clients, total of 0

Here is the logs tail from the client:

10:13:05.904 [main] INFO AkkaServer - Initial contacts: Set(akka.tcp://ICEClusterSystem@127.0.0.1:2552/system/receptionist)
10:13:05.908 [main] INFO AkkaServer - Created a cluster client actor: akka://ICEClientSystem/user/iceRequest
10:13:05.909 [main] INFO AkkaServer - Client listener started: akka://ICEClientSystem/user/clusterClientListener
[INFO] [06/01/2018 10:13:05.920] [ICEClientSystem-akka.actor.default-dispatcher-2] [akka://ICEClientSystem/deadLetters] Message [akka.cluster.client.ClusterReceptionist$Internal$GetContacts$] from Actor[akka://ICEClientSystem/user/iceRequest#-62963457] to Actor[akka://ICEClientSystem/deadLetters] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
10:13:05.929 [ICEClientSystem-akka.actor.default-dispatcher-4] INFO ClusterClientListener - ClusterClientListener: Got 1 contact points
[INFO] [06/01/2018 10:13:05.929] [ICEClientSystem-akka.actor.default-dispatcher-3] [akka://ICEClientSystem/deadLetters] Message [akka.cluster.client.ClusterReceptionist$Internal$GetContacts$] from Actor[akka://ICEClientSystem/user/iceRequest#-62963457] to Actor[akka://ICEClientSystem/deadLetters] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
10:13:05.930 [ICEClientSystem-akka.actor.default-dispatcher-4] INFO ClusterClientListener - contact: akka.tcp://ICEClusterSystem@127.0.0.1:2552/system/receptionist
[INFO] [06/01/2018 10:13:08.936] [ICEClientSystem-akka.actor.default-dispatcher-4] [akka://ICEClientSystem/deadLetters] Message [akka.cluster.client.ClusterReceptionist$Internal$GetContacts$] from Actor[akka://ICEClientSystem/user/iceRequest#-62963457] to Actor[akka://ICEClientSystem/deadLetters] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [06/01/2018 10:13:11.936] [ICEClientSystem-akka.actor.default-dispatcher-5] [akka://ICEClientSystem/deadLetters] Message [akka.cluster.client.ClusterReceptionist$Internal$GetContacts$] from Actor[akka://ICEClientSystem/user/iceRequest#-62963457] to Actor[akka://ICEClientSystem/deadLetters] was not delivered. [4] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [06/01/2018 10:13:14.937] [ICEClientSystem-akka.actor.default-dispatcher-5] [akka://ICEClientSystem/deadLetters] Message [akka.cluster.client.ClusterReceptionist$Internal$GetContacts$] from Actor[akka://ICEClientSystem/user/iceRequest#-62963457] to Actor[akka://ICEClientSystem/deadLetters] was not delivered. [5] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

The client hasn’t started sending messages yet. I noticed the same results when there is no cluster running. The client has never seen the cluster. Both are looking at 127.0.0.1:2552 for the cluster.
Any ideas?

I had a serialization class that was causing build problems. Once i fixed that correctly then cluster client worked.