Can't join cluster after upgrade to 2.6 (Artery is explicitly turned off)

Akka Libraries
1

Hi guys

I’m upgrading akka from 2.5 to 2.6 and I’m having problems with the new 2.6 nodes joining the existing 2.5 cluster. The cluster is using the kubernetes-api for service discovery, and it seems that the node is finding the cluster nodes:

{"akkaAddress":"akka.tcp://fooo@100.123.211.157:2552","akkaContactPoints":"100-118-161-100.dev.pod.cluster.local:0, 100-106-97-5.dev.pod.cluster.local:0, 100-111-91-40.dev.pod.cluster.local:0, 100-123-211-157.dev.pod.cluster.local:0","sourceThread":"foo-akka.actor.default-dispatcher-34","akkaSource":"akka://foo/system/bootstrapCoordinator","sourceActorSystem":"foo","akkaTimestamp":"10:17:21.731UTC","@timestamp":"2020-08-17T10:17:21.732+00:00","logger_name":"akka.management.cluster.bootstrap.internal.BootstrapCoordinator","thread_name":"foo-akka.actor.default-dispatcher-16","level":"INFO","message":"Located service members based on: [Lookup(foo,None,Some(tcp))]: [ResolvedTarget(100-123-211-157.dev.pod.cluster.local,None,Some(/100.123.211.157)), ResolvedTarget(100-111-91-40.dev.pod.cluster.local,None,Some(/100.111.91.40)), ResolvedTarget(100-118-161-100.dev.pod.cluster.local,None,Some(/100.118.161.100)), ResolvedTarget(100-106-97-5.dev.pod.cluster.local,None,Some(/100.106.97.5))], filtered to [100-118-161-100.dev.pod.cluster.local:0, 100-106-97-5.dev.pod.cluster.local:0, 100-111-91-40.dev.pod.cluster.local:0, 100-123-211-157.dev.pod.cluster.local:0]"}
but it fails to join the cluster:

{"akkaAddress":"akka.tcp://foo@100.123.211.157:2552","sourceThread":"foo-akka.actor.default-dispatcher-33","akkaSource":"HttpClusterBootstrapRoutes(akka://foo)","sourceActorSystem":"foo","akkaTimestamp":"10:17:22.515UTC","@timestamp":"2020-08-17T10:17:22.515+00:00","logger_name":"akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes","thread_name":"foo-akka.actor.default-dispatcher-16","level":"INFO","message":"Bootstrap request from 100.123.211.157:42702: Contact Point returning 0 seed-nodes []"}

Couldn't join seed nodes after [60] attempts, will try again. seed-nodes=[akka.tcp://foo@100.106.97.5:2552, akka.tcp://foo@100.111.91.40:2552, akka.tcp://foo@100.118.161.100:2552]

On the other nodes Im receiving a serializer exception:

Serializer not defined for message with serializer id [6] and manifest []. Transient association error (association remains live). Unknown manifest [IJ
if I perform a full cluster restart it works, but I should be able to migrate from 2.5 to 2.6 without a full restart right? Artery has been explicitly turned off. Any advice?

2

The manifest “IJ” reveals that this is related to the serializer of cluster messages, which was changed in Akka 2.6.2 and 2.6.3. https://doc.akka.io/docs/akka/current/project/rolling-update.html#2-6-2-clustermessageserializer-manifests-change
You have to update to 2.6.2 first to have compatible serializer with Akka 2.5.x. Then you can do another rollout to latest 2.6.8. I’ll add a note in the migration guide about this.

3

Great, thanks!