Hi,
I’m facing issue in akka cluster on kubernetes.
[Cluster(akka://vishnu-controller)] - Cluster Node [akka://vishnu-controller@127.0.0.1:2550] - Started up successfully
[2020-07-28 15:58:43,431] [INFO] [akka.cluster.Cluster] [vishnu-controller-akka.actor.default-dispatcher-5] [Cluster(akka://vishnu-controller)] - Cluster Node [akka://vishnu-controller@127.0.0.1:2550] - No seed-nodes configured, manual cluster join required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#joining
[2020-07-28 15:58:43,444] [INFO] [akka.cluster.sbr.SplitBrainResolver] [vishnu-controller-akka.actor.default-dispatcher-5] [akka://vishnu-controller/system/cluster/core/daemon/downingProvider] - SBR started. Config: stableAfter: 20000 ms, strategy: LeaseMajority, selfUniqueAddress: UniqueAddress(akka://vishnu-controller@127.0.0.1:2550,2743082301543251438), selfDc: default
[2020-07-28 15:58:43,834] [INFO] [akka.cluster.sharding.typed.scaladsl.ClusterSharding] [vishnu-controller-akka.actor.default-dispatcher-5] [ClusterSharding(akka://vishnu-controller)] - Starting Shard Region Proxy [Dummy] (no actors will be hosted on this node) for role [Some(write-model)] and dataCenter [None] …
[2020-07-28 15:58:43,881] [INFO] [akka.cluster.sharding.ShardRegion] [vishnu-controller-akka.actor.default-dispatcher-5] [akka://vishnu-controller@127.0.0.1:2550/system/sharding/DummyProxy] - Dummy: Idle entities will be passivated after [2.000 min]
[2020-07-28 15:58:43,908] [INFO] [akka.management.internal.HealthChecksImpl] [vishnu-controller-akka.actor.default-dispatcher-5] [HealthChecksImpl(akka://vishnu-controller)] - Loading readiness checks [(cluster-membership,akka.management.cluster.scaladsl.ClusterMembershipCheck), (example-ready,com.colortokens.vishnu.server.healthcheck.ServerHealthCheck)]
[2020-07-28 15:58:43,908] [INFO] [akka.management.internal.HealthChecksImpl] [vishnu-controller-akka.actor.default-dispatcher-5] [HealthChecksImpl(akka://vishnu-controller)] - Loading liveness checks []
[2020-07-28 15:58:43,987] [INFO] [akka.management.scaladsl.AkkaManagement] [vishnu-controller-akka.actor.default-dispatcher-5] [AkkaManagement(akka://vishnu-controller)] - Binding Akka Management (HTTP) endpoint to: 172.17.0.7:8558
[2020-07-28 15:58:44,068] [INFO] [akka.management.scaladsl.AkkaManagement] [vishnu-controller-akka.actor.default-dispatcher-5] [AkkaManagement(akka://vishnu-controller)] - Including HTTP management routes for ClusterHttpManagementRouteProvider
[2020-07-28 15:58:44,130] [INFO] [akka.management.scaladsl.AkkaManagement] [vishnu-controller-akka.actor.default-dispatcher-5] [AkkaManagement(akka://vishnu-controller)] - Including HTTP management routes for ClusterBootstrap
[2020-07-28 15:58:44,135] [INFO] [akka.management.cluster.bootstrap.ClusterBootstrap] [vishnu-controller-akka.actor.default-dispatcher-5] [ClusterBootstrap(akka://vishnu-controller)] - Using self contact point address: http://172.17.0.7:8558
[2020-07-28 15:58:44,202] [INFO] [akka.management.scaladsl.AkkaManagement] [vishnu-controller-akka.actor.default-dispatcher-5] [AkkaManagement(akka://vishnu-controller)] - Including HTTP management routes for HealthCheckRoutes
[2020-07-28 15:58:44,601] [INFO] [akka.management.cluster.bootstrap.ClusterBootstrap] [vishnu-controller-akka.actor.default-dispatcher-3] [ClusterBootstrap(akka://vishnu-controller)] - Initiating bootstrap procedure using kubernetes-api method…
[2020-07-28 15:58:44,613] [INFO] [akka.management.scaladsl.AkkaManagement] [vishnu-controller-akka.actor.default-dispatcher-5] [AkkaManagement(akka://vishnu-controller)] - Bound Akka Management (HTTP) endpoint to: 172.17.0.7:8558
[2020-07-28 15:58:44,614] [INFO] [akka.management.cluster.bootstrap.ClusterBootstrap] [vishnu-controller-akka.actor.default-dispatcher-5] [ClusterBootstrap(akka://vishnu-controller)] - Bootstrap using akka.discovery method: kubernetes-api
[2020-07-28 15:58:44,695] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-5] [akka://vishnu-controller/system/bootstrapCoordinator] - Locating service members. Using discovery [akka.discovery.kubernetes.KubernetesApiServiceDiscovery], join decider [akka.management.cluster.bootstrap.LowestAddressJoinDecider], scheme [http]
[2020-07-28 15:58:44,699] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-5] [akka://vishnu-controller/system/bootstrapCoordinator] - Looking up [Lookup(vishnu-controller,None,Some(tcp))]
[2020-07-28 15:58:44,700] [INFO] [akka.discovery.kubernetes.KubernetesApiServiceDiscovery] [vishnu-controller-akka.actor.default-dispatcher-5] [KubernetesApiServiceDiscovery(akka://vishnu-controller)] - Querying for pods with label selector: [app=vishnu-controller]. Namespace: [default]. Port: [None]
[2020-07-28 15:58:44,839] [INFO] [akka.actor.typed.ActorSystem] [scala-execution-context-global-25] [] - GRPC server started and bound to port : ServerBinding(/127.0.0.1:8080)
[2020-07-28 15:58:45,789] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-3] [akka://vishnu-controller/system/bootstrapCoordinator] - Looking up [Lookup(vishnu-controller,None,Some(tcp))]
[2020-07-28 15:58:45,790] [INFO] [akka.discovery.kubernetes.KubernetesApiServiceDiscovery] [vishnu-controller-akka.actor.default-dispatcher-3] [KubernetesApiServiceDiscovery(akka://vishnu-controller)] - Querying for pods with label selector: [app=vishnu-controller]. Namespace: [default]. Port: [None]
[2020-07-28 15:58:46,236] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-11] [akka://vishnu-controller/system/bootstrapCoordinator] - Located service members based on: [Lookup(vishnu-controller,None,Some(tcp))]: [ResolvedTarget(172-17-0-9.default.pod.cluster.local,None,Some(/172.17.0.9)), ResolvedTarget(172-17-0-7.default.pod.cluster.local,None,Some(/172.17.0.7))], filtered to [172-17-0-9.default.pod.cluster.local:0, 172-17-0-7.default.pod.cluster.local:0]
[2020-07-28 15:58:46,329] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-23] [akka://vishnu-controller/system/bootstrapCoordinator] - Located service members based on: [Lookup(vishnu-controller,None,Some(tcp))]: [ResolvedTarget(172-17-0-9.default.pod.cluster.local,None,Some(/172.17.0.9)), ResolvedTarget(172-17-0-7.default.pod.cluster.local,None,Some(/172.17.0.7))], filtered to [172-17-0-9.default.pod.cluster.local:0, 172-17-0-7.default.pod.cluster.local:0]
[2020-07-28 15:58:46,544] [WARN] [akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap] [vishnu-controller-akka.actor.default-dispatcher-21] [akka://vishnu-controller@127.0.0.1:2550/system/bootstrapCoordinator/contactPointProbe-172-17-0-9.default.pod.cluster.local-8558] - Probing [http://172-17-0-9.default.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Tcp command [Connect(172-17-0-9.default.pod.cluster.local:8558,None,List(),Some(10 seconds),true)] failed because of java.net.ConnectException: Connection refused
[2020-07-28 15:58:46,648] [INFO] [akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes] [vishnu-controller-akka.actor.default-dispatcher-21] [HttpClusterBootstrapRoutes(akka://vishnu-controller)] - Bootstrap request from 172.17.0.7:39274: Contact Point returning 0 seed-nodes []
[2020-07-28 15:58:46,692] [INFO] [akka.management.cluster.bootstrap.LowestAddressJoinDecider] [vishnu-controller-akka.actor.default-dispatcher-5] [LowestAddressJoinDecider(akka://vishnu-controller)] - Discovered [2] contact points, confirmed [0], which is less than the required [2], retrying
[2020-07-28 15:58:46,880] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-21] [akka://vishnu-controller/system/bootstrapCoordinator] - Contact point [akka://vishnu-controller@127.0.0.1:2550] returned [0] seed-nodes []
[2020-07-28 15:58:47,553] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-13] [akka://vishnu-controller/system/bootstrapCoordinator] - Looking up [Lookup(vishnu-controller,None,Some(tcp))]
[2020-07-28 15:58:47,553] [INFO] [akka.discovery.kubernetes.KubernetesApiServiceDiscovery] [vishnu-controller-akka.actor.default-dispatcher-13] [KubernetesApiServiceDiscovery(akka://vishnu-controller)] - Querying for pods with label selector: [app=vishnu-controller]. Namespace: [default]. Port: [None]
[2020-07-28 15:58:47,591] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-23] [akka://vishnu-controller/system/bootstrapCoordinator] - Located service members based on: [Lookup(vishnu-controller,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8)), ResolvedTarget(172-17-0-9.default.pod.cluster.local,None,Some(/172.17.0.9)), ResolvedTarget(172-17-0-7.default.pod.cluster.local,None,Some(/172.17.0.7))], filtered to [172-17-0-9.default.pod.cluster.local:0, 172-17-0-8.default.pod.cluster.local:0, 172-17-0-7.default.pod.cluster.local:0]
[2020-07-28 15:58:47,652] [WARN] [akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap] [vishnu-controller-akka.actor.default-dispatcher-21] [akka://vishnu-controller@127.0.0.1:2550/system/bootstrapCoordinator/contactPointProbe-172-17-0-8.default.pod.cluster.local-8558] - Probing [http://172-17-0-8.default.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Tcp command [Connect(172-17-0-8.default.pod.cluster.local:8558,None,List(),Some(10 seconds),true)] failed because of java.net.ConnectException: Connection refused
[2020-07-28 15:58:47,702] [INFO] [akka.management.cluster.bootstrap.LowestAddressJoinDecider] [vishnu-controller-akka.actor.default-dispatcher-5] [LowestAddressJoinDecider(akka://vishnu-controller)] - Discovered [3] contact points, confirmed [1], which is less than the required [2], retrying
[2020-07-28 15:58:47,705] [WARN] [akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap] [vishnu-controller-akka.actor.default-dispatcher-23] [akka://vishnu-controller@127.0.0.1:2550/system/bootstrapCoordinator/contactPointProbe-172-17-0-9.default.pod.cluster.local-8558] - Probing [http://172-17-0-9.default.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Tcp command [Connect(172-17-0-9.default.pod.cluster.local:8558,None,List(),Some(10 seconds),true)] failed because of java.net.ConnectException: Connection refused
[2020-07-28 15:58:47,983] [INFO] [akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes] [vishnu-controller-akka.actor.default-dispatcher-21] [HttpClusterBootstrapRoutes(akka://vishnu-controller)] - Bootstrap request from 172.17.0.7:39274: Contact Point returning 0 seed-nodes []
[2020-07-28 15:58:47,984] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-5] [akka://vishnu-controller/system/bootstrapCoordinator] - Contact point [akka://vishnu-controller@127.0.0.1:2550] returned [0] seed-nodes []
[2020-07-28 15:58:48,701] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-12] [akka://vishnu-controller/system/bootstrapCoordinator] - Looking up [Lookup(vishnu-controller,None,Some(tcp))]
[2020-07-28 15:58:48,701] [INFO] [akka.discovery.kubernetes.KubernetesApiServiceDiscovery] [vishnu-controller-akka.actor.default-dispatcher-12] [KubernetesApiServiceDiscovery(akka://vishnu-controller)] - Querying for pods with label selector: [app=vishnu-controller]. Namespace: [default]. Port: [None]
[2020-07-28 15:58:48,710] [INFO] [akka.management.cluster.bootstrap.LowestAddressJoinDecider] [vishnu-controller-akka.actor.default-dispatcher-5] [LowestAddressJoinDecider(akka://vishnu-controller)] - Discovered [3] contact points, confirmed [1], which is less than the required [2], retrying
[2020-07-28 15:58:48,716] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-21] [akka://vishnu-controller/system/bootstrapCoordinator] - Located service members based on: [Lookup(vishnu-controller,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8)), ResolvedTarget(172-17-0-9.default.pod.cluster.local,None,Some(/172.17.0.9)), ResolvedTarget(172-17-0-7.default.pod.cluster.local,None,Some(/172.17.0.7))], filtered to [172-17-0-9.default.pod.cluster.local:0, 172-17-0-8.default.pod.cluster.local:0, 172-17-0-7.default.pod.cluster.local:0]
[2020-07-28 15:58:48,793] [WARN] [akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap] [vishnu-controller-akka.actor.default-dispatcher-23] [akka://vishnu-controller@127.0.0.1:2550/system/bootstrapCoordinator/contactPointProbe-172-17-0-9.default.pod.cluster.local-8558] - Probing [http://172-17-0-9.default.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Tcp command [Connect(172-17-0-9.default.pod.cluster.local:8558,None,List(),Some(10 seconds),true)] failed because of java.net.ConnectException: Connection refused
[2020-07-28 15:58:48,841] [WARN] [akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap] [vishnu-controller-akka.actor.default-dispatcher-22] [akka://vishnu-controller@127.0.0.1:2550/system/bootstrapCoordinator/contactPointProbe-172-17-0-8.default.pod.cluster.local-8558] - Probing [http://172-17-0-8.default.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Tcp command [Connect(172-17-0-8.default.pod.cluster.local:8558,None,List(),Some(10 seconds),true)] failed because of java.net.ConnectException: Connection refused
[2020-07-28 15:58:49,093] [INFO] [akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes] [vishnu-controller-akka.actor.default-dispatcher-22] [HttpClusterBootstrapRoutes(akka://vishnu-controller)] - Bootstrap request from 172.17.0.7:39274: Contact Point returning 0 seed-nodes []
[2020-07-28 15:58:49,099] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-22] [akka://vishnu-controller/system/bootstrapCoordinator] - Contact point [akka://vishnu-controller@127.0.0.1:2550] returned [0] seed-nodes []
[2020-07-28 15:58:49,700] [INFO] [akka.management.cluster.bootstrap.LowestAddressJoinDecider] [vishnu-controller-akka.actor.default-dispatcher-5] [LowestAddressJoinDecider(akka://vishnu-controller)] - Discovered [3] contact points, confirmed [1], which is less than the required [2], retrying
[2020-07-28 15:58:49,881] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-11] [akka://vishnu-controller/system/bootstrapCoordinator] - Looking up [Lookup(vishnu-controller,None,Some(tcp))]
[2020-07-28 15:58:49,882] [INFO] [akka.discovery.kubernetes.KubernetesApiServiceDiscovery] [vishnu-controller-akka.actor.default-dispatcher-11] [KubernetesApiServiceDiscovery(akka://vishnu-controller)] - Querying for pods with label selector: [app=vishnu-controller]. Namespace: [default]. Port: [None]
[2020-07-28 15:58:49,894] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-21] [akka://vishnu-controller/system/bootstrapCoordinator] - Located service members based on: [Lookup(vishnu-controller,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8)), ResolvedTarget(172-17-0-9.default.pod.cluster.local,None,Some(/172.17.0.9)), ResolvedTarget(172-17-0-7.default.pod.cluster.local,None,Some(/172.17.0.7))], filtered to [172-17-0-9.default.pod.cluster.local:0, 172-17-0-8.default.pod.cluster.local:0, 172-17-0-7.default.pod.cluster.local:0]
[2020-07-28 15:58:50,025] [WARN] [akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap] [vishnu-controller-akka.actor.default-dispatcher-11] [akka://vishnu-controller@127.0.0.1:2550/system/bootstrapCoordinator/contactPointProbe-172-17-0-8.default.pod.cluster.local-8558] - Probing [http://172-17-0-8.default.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Tcp command [Connect(172-17-0-8.default.pod.cluster.local:8558,None,List(),Some(10 seconds),true)] failed because of java.net.ConnectException: Connection refused
[2020-07-28 15:58:50,215] [INFO] [akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes] [vishnu-controller-akka.actor.default-dispatcher-21] [HttpClusterBootstrapRoutes(akka://vishnu-controller)] - Bootstrap request from 172.17.0.7:39274: Contact Point returning 0 seed-nodes []
[2020-07-28 15:58:50,241] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-21] [akka://vishnu-controller/system/bootstrapCoordinator] - Contact point [akka://vishnu-controller@127.0.0.1:2550] returned [0] seed-nodes []
[2020-07-28 15:58:50,673] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-23] [akka://vishnu-controller/system/bootstrapCoordinator] - Contact point [akka://vishnu-controller@127.0.0.1:2550] returned [0] seed-nodes []
[2020-07-28 15:58:51,040] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-23] [akka://vishnu-controller/system/bootstrapCoordinator] - Looking up [Lookup(vishnu-controller,None,Some(tcp))]
[2020-07-28 15:58:51,041] [INFO] [akka.discovery.kubernetes.KubernetesApiServiceDiscovery] [vishnu-controller-akka.actor.default-dispatcher-23] [KubernetesApiServiceDiscovery(akka://vishnu-controller)] - Querying for pods with label selector: [app=vishnu-controller]. Namespace: [default]. Port: [None]
[2020-07-28 15:58:51,087] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-22] [akka://vishnu-controller/system/bootstrapCoordinator] - Located service members based on: [Lookup(vishnu-controller,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8)), ResolvedTarget(172-17-0-9.default.pod.cluster.local,None,Some(/172.17.0.9)), ResolvedTarget(172-17-0-7.default.pod.cluster.local,None,Some(/172.17.0.7))], filtered to [172-17-0-9.default.pod.cluster.local:0, 172-17-0-8.default.pod.cluster.local:0, 172-17-0-7.default.pod.cluster.local:0]
[2020-07-28 15:58:51,143] [WARN] [akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap] [vishnu-controller-akka.actor.default-dispatcher-22] [akka://vishnu-controller@127.0.0.1:2550/system/bootstrapCoordinator/contactPointProbe-172-17-0-8.default.pod.cluster.local-8558] - Probing [http://172-17-0-8.default.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Tcp command [Connect(172-17-0-8.default.pod.cluster.local:8558,None,List(),Some(10 seconds),true)] failed because of java.net.ConnectException: Connection refused
[2020-07-28 15:58:51,147] [ERROR] [akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap] [vishnu-controller-akka.actor.default-dispatcher-22] [akka://vishnu-controller@127.0.0.1:2550/system/bootstrapCoordinator/contactPointProbe-172-17-0-8.default.pod.cluster.local-8558] - Overdue of probing-failure-timeout, stop probing, signaling that it’s failed
[2020-07-28 15:58:51,150] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-22] [akka://vishnu-controller/system/bootstrapCoordinator] - Received signal that probing has failed, scheduling contact point probing again
[2020-07-28 15:58:51,395] [INFO] [akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes] [vishnu-controller-akka.actor.default-dispatcher-5] [HttpClusterBootstrapRoutes(akka://vishnu-controller)] - Bootstrap request from 172.17.0.7:39274: Contact Point returning 0 seed-nodes []
[2020-07-28 15:58:51,399] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-5] [akka://vishnu-controller/system/bootstrapCoordinator] - Contact point [akka://vishnu-controller@127.0.0.1:2550] returned [0] seed-nodes []
[2020-07-28 15:58:51,889] [INFO] [akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes] [vishnu-controller-akka.actor.default-dispatcher-3] [HttpClusterBootstrapRoutes(akka://vishnu-controller)] - Bootstrap request from 172.17.0.9:47462: Contact Point returning 0 seed-nodes []
[2020-07-28 15:58:51,896] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-3] [akka://vishnu-controller/system/bootstrapCoordinator] - Contact point [akka://vishnu-controller@127.0.0.1:2550] returned [0] seed-nodes []
[2020-07-28 15:58:52,224] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-3] [akka://vishnu-controller/system/bootstrapCoordinator] - Looking up [Lookup(vishnu-controller,None,Some(tcp))]
[2020-07-28 15:58:52,225] [INFO] [akka.discovery.kubernetes.KubernetesApiServiceDiscovery] [vishnu-controller-akka.actor.default-dispatcher-3] [KubernetesApiServiceDiscovery(akka://vishnu-controller)] - Querying for pods with label selector: [app=vishnu-controller]. Namespace: [default]. Port: [None]
[2020-07-28 15:58:52,239] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-5] [akka://vishnu-controller/system/bootstrapCoordinator] - Located service members based on: [Lookup(vishnu-controller,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8)), ResolvedTarget(172-17-0-9.default.pod.cluster.local,None,Some(/172.17.0.9)), ResolvedTarget(172-17-0-7.default.pod.cluster.local,None,Some(/172.17.0.7))], filtered to [172-17-0-9.default.pod.cluster.local:0, 172-17-0-8.default.pod.cluster.local:0, 172-17-0-7.default.pod.cluster.local:0]
[2020-07-28 15:58:52,265] [WARN] [akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap] [vishnu-controller-akka.actor.default-dispatcher-22] [akka://vishnu-controller@127.0.0.1:2550/system/bootstrapCoordinator/contactPointProbe-172-17-0-8.default.pod.cluster.local-8558] - Probing [http://172-17-0-8.default.pod.cluster.local:8558/bootstrap/seed-nodes] failed due to: Tcp command [Connect(172-17-0-8.default.pod.cluster.local:8558,None,List(),Some(10 seconds),true)] failed because of java.net.ConnectException: Connection refused
[2020-07-28 15:58:52,492] [INFO] [akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes] [vishnu-controller-akka.actor.default-dispatcher-13] [HttpClusterBootstrapRoutes(akka://vishnu-controller)] - Bootstrap request from 172.17.0.7:39274: Contact Point returning 0 seed-nodes []
[2020-07-28 15:58:52,497] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-13] [akka://vishnu-controller/system/bootstrapCoordinator] - Contact point [akka://vishnu-controller@127.0.0.1:2550] returned [0] seed-nodes []
[2020-07-28 15:58:52,701] [INFO] [akka.management.cluster.bootstrap.LowestAddressJoinDecider] [vishnu-controller-akka.actor.default-dispatcher-11] [LowestAddressJoinDecider(akka://vishnu-controller)] - Exceeded stable margins but missing seed node information from some contact points [172-17-0-8.default.pod.cluster.local:0] (out of [172-17-0-9.default.pod.cluster.local:0, 172-17-0-8.default.pod.cluster.local:0, 172-17-0-7.default.pod.cluster.local:0])
[2020-07-28 15:58:53,081] [INFO] [akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes] [vishnu-controller-akka.actor.default-dispatcher-11] [HttpClusterBootstrapRoutes(akka://vishnu-controller)] - Bootstrap request from 172.17.0.9:47462: Contact Point returning 0 seed-nodes []
[2020-07-28 15:58:53,118] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-11] [akka://vishnu-controller/system/bootstrapCoordinator] - Contact point [akka://vishnu-controller@127.0.0.1:2550] returned [0] seed-nodes []
[2020-07-28 15:58:53,191] [INFO] [akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes] [vishnu-controller-akka.actor.default-dispatcher-11] [HttpClusterBootstrapRoutes(akka://vishnu-controller)] - Bootstrap request from 172.17.0.8:49952: Contact Point returning 0 seed-nodes []
[2020-07-28 15:58:53,424] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-13] [akka://vishnu-controller/system/bootstrapCoordinator] - Looking up [Lookup(vishnu-controller,None,Some(tcp))]
[2020-07-28 15:58:53,425] [INFO] [akka.discovery.kubernetes.KubernetesApiServiceDiscovery] [vishnu-controller-akka.actor.default-dispatcher-13] [KubernetesApiServiceDiscovery(akka://vishnu-controller)] - Querying for pods with label selector: [app=vishnu-controller]. Namespace: [default]. Port: [None]
[2020-07-28 15:58:53,458] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-3] [akka://vishnu-controller/system/bootstrapCoordinator] - Located service members based on: [Lookup(vishnu-controller,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8)), ResolvedTarget(172-17-0-9.default.pod.cluster.local,None,Some(/172.17.0.9)), ResolvedTarget(172-17-0-7.default.pod.cluster.local,None,Some(/172.17.0.7))], filtered to [172-17-0-9.default.pod.cluster.local:0, 172-17-0-8.default.pod.cluster.local:0, 172-17-0-7.default.pod.cluster.local:0]
[2020-07-28 15:58:53,684] [INFO] [akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes] [vishnu-controller-akka.actor.default-dispatcher-13] [HttpClusterBootstrapRoutes(akka://vishnu-controller)] - Bootstrap request from 172.17.0.7:39274: Contact Point returning 0 seed-nodes []
[2020-07-28 15:58:53,685] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-23] [akka://vishnu-controller/system/bootstrapCoordinator] - Contact point [akka://vishnu-controller@127.0.0.1:2550] returned [0] seed-nodes []
[2020-07-28 15:58:53,699] [INFO] [akka.management.cluster.bootstrap.LowestAddressJoinDecider] [vishnu-controller-akka.actor.default-dispatcher-11] [LowestAddressJoinDecider(akka://vishnu-controller)] - Exceeded stable margins but missing seed node information from some contact points [172-17-0-8.default.pod.cluster.local:0] (out of [172-17-0-9.default.pod.cluster.local:0, 172-17-0-8.default.pod.cluster.local:0, 172-17-0-7.default.pod.cluster.local:0])
[2020-07-28 15:58:54,018] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-12] [akka://vishnu-controller/system/bootstrapCoordinator] - Contact point [akka://vishnu-controller@127.0.0.1:2550] returned [0] seed-nodes []
[2020-07-28 15:58:54,275] [INFO] [akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes] [vishnu-controller-akka.actor.default-dispatcher-3] [HttpClusterBootstrapRoutes(akka://vishnu-controller)] - Bootstrap request from 172.17.0.9:47462: Contact Point returning 0 seed-nodes []
[2020-07-28 15:58:54,336] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-3] [akka://vishnu-controller/system/bootstrapCoordinator] - Contact point [akka://vishnu-controller@127.0.0.1:2550] returned [0] seed-nodes []
[2020-07-28 15:58:54,373] [INFO] [akka.management.cluster.bootstrap.contactpoint.HttpClusterBootstrapRoutes] [vishnu-controller-akka.actor.default-dispatcher-11] [HttpClusterBootstrapRoutes(akka://vishnu-controller)] - Bootstrap request from 172.17.0.8:49952: Contact Point returning 0 seed-nodes []
[2020-07-28 15:58:54,623] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-11] [akka://vishnu-controller/system/bootstrapCoordinator] - Looking up [Lookup(vishnu-controller,None,Some(tcp))]
[2020-07-28 15:58:54,623] [INFO] [akka.discovery.kubernetes.KubernetesApiServiceDiscovery] [vishnu-controller-akka.actor.default-dispatcher-11] [KubernetesApiServiceDiscovery(akka://vishnu-controller)] - Querying for pods with label selector: [app=vishnu-controller]. Namespace: [default]. Port: [None]
[2020-07-28 15:58:54,661] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-3] [akka://vishnu-controller/system/bootstrapCoordinator] - Located service members based on: [Lookup(vishnu-controller,None,Some(tcp))]: [ResolvedTarget(172-17-0-8.default.pod.cluster.local,None,Some(/172.17.0.8)), ResolvedTarget(172-17-0-9.default.pod.cluster.local,None,Some(/172.17.0.9)), ResolvedTarget(172-17-0-7.default.pod.cluster.local,None,Some(/172.17.0.7))], filtered to [172-17-0-9.default.pod.cluster.local:0, 172-17-0-8.default.pod.cluster.local:0, 172-17-0-7.default.pod.cluster.local:0]
[2020-07-28 15:58:54,702] [INFO] [akka.management.cluster.bootstrap.internal.BootstrapCoordinator] [vishnu-controller-akka.actor.default-dispatcher-5] [akka://vishnu-controller/system/bootstrapCoordinator] - Initiating new cluster, self-joining [akka://vishnu-controller@127.0.0.1:2550]. Other nodes are expected to locate this cluster via continued contact-point probing.
[2020-07-28 15:58:54,710] [INFO] [akka.remote.RemoteActorRefProvider$RemoteDeadLetterActorRef] [vishnu-controller-akka.actor.default-dispatcher-12] [akka://vishnu-controller/deadLetters] - Message [akka.management.cluster.bootstrap.internal.BootstrapCoordinator$Protocol$BootstrappingCompleted$] from Actor[akka://vishnu-controller/system/bootstrapCoordinator#-1611751782] to Actor[akka://vishnu-controller/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior then Actor[akka://vishnu-controller/deadLetters] may have terminated unexpectedly. This logging can be turned off or adjusted with configuration settings ‘akka.log-dead-letters’ and ‘akka.log-dead-letters-during-shutdown’.
[2020-07-28 15:58:54,714] [INFO] [akka.cluster.Cluster] [vishnu-controller-akka.actor.default-dispatcher-12] [Cluster(akka://vishnu-controller)] - Cluster Node [akka://vishnu-controller@127.0.0.1:2550] - Node [akka://vishnu-controller@127.0.0.1:2550] is JOINING itself (with roles [dc-default]) and forming new cluster
[2020-07-28 15:58:54,714] [INFO] [akka.cluster.Cluster] [vishnu-controller-akka.actor.default-dispatcher-12] [Cluster(akka://vishnu-controller)] - Cluster Node [akka://vishnu-controller@127.0.0.1:2550] - is the new leader among reachable nodes (more leaders may exist)
[2020-07-28 15:58:54,729] [INFO] [akka.cluster.Cluster] [vishnu-controller-akka.actor.default-dispatcher-3] [Cluster(akka://vishnu-controller)] - Cluster Node [akka://vishnu-controller@127.0.0.1:2550] - Leader is moving node [akka://vishnu-controller@127.0.0.1:2550] to [Up]
[2020-07-28 15:58:54,736] [INFO] [com.colortokens.vishnu.server.infrastructure.Guardian$] [vishnu-controller-akka.actor.default-dispatcher-3] [akka://vishnu-controller/user/listener] - MemberEvent: MemberUp(Member(address = akka://vishnu-controller@127.0.0.1:2550, status = Up))
[2020-07-28 15:58:54,744] [INFO] [akka.cluster.sbr.SplitBrainResolver] [vishnu-controller-akka.actor.default-dispatcher-5] [akka://vishnu-controller/system/cluster/core/daemon/downingProvider] - This node is now the leader responsible for taking SBR decisions among the reachable nodes (more leaders may exist).