Hello:
I am new to Lagom/Akka and trying to learn while doing. I have an app with three services that I am trying to run Docker via docker-compose.
The leader node seems to start just fine, no glaring warnings. Both non-leader members do not. Log below:
2020-04-21T19:52:44.121Z [info] akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
2020-04-21T19:52:44.609Z [info] akka.remote.artery.tcp.ArteryTcpTransport [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=ArteryTcpTransport(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:44.608UTC] - Remoting started with transport [Artery tcp]; listening on address [akka://application@app-datasource:25520] with UID [8651354229357041999]
2020-04-21T19:52:44.641Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:44.641UTC] - Cluster Node [akka://application@app-datasource:25520] - Starting up, Akka version [2.6.3] ...
2020-04-21T19:52:44.813Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:44.813UTC] - Cluster Node [akka://application@app-datasource:25520] - Registered cluster JMX MBean [akka:type=Cluster]
2020-04-21T19:52:44.813Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:44.813UTC] - Cluster Node [akka://application@app-datasource:25520] - Started up successfully
2020-04-21T19:52:44.872Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-4, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:44.872UTC] - Cluster Node [akka://application@app-datasource:25520] - No downing-provider-class configured, manual cluster downing required, see https://doc.akka.io/docs/akka/current/typed/cluster.html#downing
2020-04-21T19:52:46.192Z [info] akka.management.cluster.bootstrap.ClusterBootstrap [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=ClusterBootstrap(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.190UTC] - Bootstrap using `akka.discovery` method: akka-dns
2020-04-21T19:52:46.212Z [info] akka.io.DnsExt [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=DnsExt(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.212UTC] - Creating async dns resolver async-dns with manager name SD-DNS
2020-04-21T19:52:46.280Z [info] akka.management.internal.HealthChecksImpl [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=HealthChecksImpl(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.280UTC] - Loading readiness checks List(NamedHealthCheck(cluster-membership,akka.management.cluster.scaladsl.ClusterMembershipCheck))
2020-04-21T19:52:46.280Z [info] akka.management.internal.HealthChecksImpl [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=HealthChecksImpl(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.280UTC] - Loading liveness checks List()
2020-04-21T19:52:46.395Z [warn] akka.stream.Materializer [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.remote.default-remote-dispatcher-10, akkaSource=akka.stream.Log(akka://application/system/Materializers/StreamSupervisor-1), sourceActorSystem=application, akkaTimestamp=19:52:46.390UTC] - [outbound connection to [akka://application@app-process:25520], control stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(app-process:25520,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused
2020-04-21T19:52:46.395Z [warn] akka.stream.Materializer [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.remote.default-remote-dispatcher-6, akkaSource=akka.stream.Log(akka://application/system/Materializers/StreamSupervisor-1), sourceActorSystem=application, akkaTimestamp=19:52:46.392UTC] - [outbound connection to [akka://application@app-process:25520], message stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(app-process:25520,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused
2020-04-21T19:52:46.404Z [warn] akka.stream.Materializer [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.remote.default-remote-dispatcher-12, akkaSource=akka.stream.Log(akka://application/system/Materializers/StreamSupervisor-1), sourceActorSystem=application, akkaTimestamp=19:52:46.392UTC] - [outbound connection to [akka://application@test:25520], control stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(test:25520,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused
2020-04-21T19:52:46.416Z [warn] akka.stream.Materializer [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.remote.default-remote-dispatcher-8, akkaSource=akka.stream.Log(akka://application/system/Materializers/StreamSupervisor-1), sourceActorSystem=application, akkaTimestamp=19:52:46.391UTC] - [outbound connection to [akka://application@test:25520], message stream] Upstream failed, cause: StreamTcpException: Tcp command [Connect(test:25520,None,List(),Some(5000 milliseconds),true)] failed because of java.net.ConnectException: Connection refused
2020-04-21T19:52:46.497Z [info] akka.management.scaladsl.AkkaManagement [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=AkkaManagement(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.496UTC] - Binding Akka Management (HTTP) endpoint to: app-datasource:8558
2020-04-21T19:52:46.586Z [info] akka.management.scaladsl.AkkaManagement [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=AkkaManagement(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.586UTC] - Including HTTP management routes for ClusterHttpManagementRouteProvider
2020-04-21T19:52:46.667Z [info] akka.management.scaladsl.AkkaManagement [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=AkkaManagement(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.665UTC] - Including HTTP management routes for ClusterBootstrap
2020-04-21T19:52:46.673Z [info] akka.management.cluster.bootstrap.ClusterBootstrap [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=ClusterBootstrap(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.673UTC] - Using self contact point address: http://app-datasource:8558
2020-04-21T19:52:46.698Z [info] akka.management.scaladsl.AkkaManagement [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=AkkaManagement(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:46.698UTC] - Including HTTP management routes for HealthCheckRoutes
2020-04-21T19:52:47.616Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-13, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:47.616UTC] - Cluster Node [akka://application@app-datasource:25520] - Received InitJoin message from [Actor[akka://application@app-cron:25520/system/cluster/core/daemon/firstSeedNodeProcess-1#97626252]], but this node is not initialized yet
2020-04-21T19:52:47.619Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-13, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:47.619UTC] - Cluster Node [akka://application@app-datasource:25520] - Received InitJoin message from [Actor[akka://application@app-cron:25520/system/cluster/core/daemon/firstSeedNodeProcess-1#97626252]], but this node is not initialized yet
2020-04-21T19:52:47.736Z [info] akka.management.scaladsl.AkkaManagement [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.default-dispatcher-32, akkaSource=AkkaManagement(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:47.736UTC] - Bound Akka Management (HTTP) endpoint to: app-datasource:8558
2020-04-21T19:52:48.800Z [info] akka.cluster.sharding.ShardRegion [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-22, akkaSource=akka://application@app-datasource:25520/system/sharding/kafkaProducer-app_datasource, sourceActorSystem=application, akkaTimestamp=19:52:48.800UTC] - kafkaProducer-app_datasource: Idle entities will be passivated after [2.000 min]
2020-04-21T19:52:48.808Z [info] akka.cluster.sharding.typed.scaladsl.ClusterSharding [akkaAddress=akka://application@app-datasource:25520, sourceThread=main, akkaSource=ClusterSharding(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:48.808UTC] - Starting Shard Region [DataSource]...
2020-04-21T19:52:48.814Z [info] akka.cluster.sharding.ShardRegion [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-15, akkaSource=akka://application@app-datasource:25520/system/sharding/DataSource, sourceActorSystem=application, akkaTimestamp=19:52:48.814UTC] - DataSource: Idle entities will be passivated after [2.000 min]
2020-04-21T19:52:48.921Z [info] play.api.Play [] - Application started (Prod) (no global state)
2020-04-21T19:52:48.961Z [info] play.core.server.AkkaHttpServer [] - Listening for HTTP on /0.0.0.0:9000
2020-04-21T19:52:49.395Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-14, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:49.394UTC] - Cluster Node [akka://application@app-datasource:25520] - Received InitJoin message from [Actor[akka://application@app-process:25520/system/cluster/core/daemon/joinSeedNodeProcess-1#618986798]], but this node is not initialized yet
2020-04-21T19:52:49.414Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-15, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:49.414UTC] - Cluster Node [akka://application@app-datasource:25520] - Received InitJoin message from [Actor[akka://application@test:25520/system/cluster/core/daemon/joinSeedNodeProcess-1#724457852]], but this node is not initialized yet
2020-04-21T19:52:54.496Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-22, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:54.496UTC] - Cluster Node [akka://application@app-datasource:25520] - Received InitJoin message from [Actor[akka://application@app-process:25520/system/cluster/core/daemon/joinSeedNodeProcess-1#618986798]], but this node is not initialized yet
2020-04-21T19:52:54.503Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-21, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:54.502UTC] - Cluster Node [akka://application@app-datasource:25520] - Received InitJoin message from [Actor[akka://application@test:25520/system/cluster/core/daemon/joinSeedNodeProcess-1#724457852]], but this node is not initialized yet
2020-04-21T19:52:54.638Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-3, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:54.638UTC] - Cluster Node [akka://application@app-datasource:25520] - Received InitJoinAck message from [Actor[akka://application@app-cron:25520/system/cluster/core/daemon#193428041]] to [akka://application@app-datasource:25520]
2020-04-21T19:52:55.494Z [info] akka.cluster.Cluster [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-23, akkaSource=Cluster(akka://application), sourceActorSystem=application, akkaTimestamp=19:52:55.493UTC] - Cluster Node [akka://application@app-datasource:25520] - Welcome from [akka://application@app-cron:25520]
2020-04-21T19:52:55.814Z [info] akka.cluster.singleton.ClusterSingletonManager [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-22, akkaSource=akka://application@app-datasource:25520/system/sharding/DataSourceCoordinator, sourceActorSystem=application, akkaTimestamp=19:52:55.814UTC] - ClusterSingletonManager state change [Start -> Younger]
2020-04-21T19:52:55.816Z [info] akka.cluster.singleton.ClusterSingletonManager [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-39, akkaSource=akka://application@app-datasource:25520/user/$a, sourceActorSystem=application, akkaTimestamp=19:52:55.814UTC] - ClusterSingletonManager state change [Start -> Younger]
2020-04-21T19:52:55.819Z [info] akka.cluster.singleton.ClusterSingletonManager [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-23, akkaSource=akka://application@app-datasource:25520/user/cassandraOffsetStorePrepare-singleton, sourceActorSystem=application, akkaTimestamp=19:52:55.817UTC] - ClusterSingletonManager state change [Start -> Younger]
2020-04-21T19:52:55.819Z [info] akka.cluster.singleton.ClusterSingletonManager [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-4, akkaSource=akka://application@app-datasource:25520/system/sharding/kafkaProducer-app_datasourceCoordinator, sourceActorSystem=application, akkaTimestamp=19:52:55.818UTC] - ClusterSingletonManager state change [Start -> Younger]
2020-04-21T19:52:56.318Z [info] akka.cluster.singleton.ClusterSingletonProxy [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-23, akkaSource=akka://application@app-datasource:25520/user/cassandraOffsetStorePrepare-singletonProxy, sourceActorSystem=application, akkaTimestamp=19:52:56.317UTC] - Singleton identified at [akka://application@app-cron:25520/user/cassandraOffsetStorePrepare-singleton/singleton]
2020-04-21T19:52:58.893Z [warn] akka.cluster.sharding.ShardRegion [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-33, akkaSource=akka://application@app-datasource:25520/system/sharding/kafkaProducer-app_datasource, sourceActorSystem=application, akkaTimestamp=19:52:58.893UTC] - kafkaProducer-app_datasource: Trying to register to coordinator at [ActorSelection[Anchor(akka://application@app-cron:25520/), Path(/system/sharding/kafkaProducer-app_datasourceCoordinator/singleton/coordinator)]], but no acknowledgement. Total [1] buffered messages. [Coordinator [Member(address = akka://application@app-cron:25520, status = Up)] is reachable.]
2020-04-21T19:53:00.913Z [warn] akka.cluster.sharding.ShardRegion [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-14, akkaSource=akka://application@app-datasource:25520/system/sharding/kafkaProducer-app_datasource, sourceActorSystem=application, akkaTimestamp=19:53:00.912UTC] - kafkaProducer-app_datasource: Trying to register to coordinator at [ActorSelection[Anchor(akka://application@app-cron:25520/), Path(/system/sharding/kafkaProducer-app_datasourceCoordinator/singleton/coordinator)]], but no acknowledgement. Total [1] buffered messages. [Coordinator [Member(address = akka://application@app-cron:25520, status = Up)] is reachable.]
2020-04-21T19:53:02.933Z [warn] akka.cluster.sharding.ShardRegion [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-33, akkaSource=akka://application@app-datasource:25520/system/sharding/kafkaProducer-app_datasource, sourceActorSystem=application, akkaTimestamp=19:53:02.933UTC] - kafkaProducer-app_datasource: Trying to register to coordinator at [ActorSelection[Anchor(akka://application@app-cron:25520/), Path(/system/sharding/kafkaProducer-app_datasourceCoordinator/singleton/coordinator)]], but no acknowledgement. Total [1] buffered messages. [Coordinator [Member(address = akka://application@app-cron:25520, status = Up)] is reachable.]
2020-04-21T19:53:04.953Z [warn] akka.cluster.sharding.ShardRegion [akkaAddress=akka://application@app-datasource:25520, sourceThread=application-akka.actor.internal-dispatcher-13, akkaSource=akka://application@app-datasource:25520/system/sharding/kafkaProducer-app_datasource, sourceActorSystem=application, akkaTimestamp=19:53:04.952UTC] - kafkaProducer-app_datasource: Trying to register to coordinator at [ActorSelection[Anchor(akka://application@app-cron:25520/), Path(/system/sharding/kafkaProducer-app_datasourceCoordinator/singleton/coordinator)]], but no acknowledgement. Total [1] buffered messages. [Coordinator [Member(address = akka://application@app-cron:25520, status = Up)] is reachable.]
The configuration:
play.application.loader = com.example.app.datasource.impl.DataStoreLoader
app.cassandra.keyspace = app_datasource
cassandra-journal.keyspace = ${app.cassandra.keyspace}
cassandra-snapshot-store.keyspace = ${app.cassandra.keyspace}
lagom.persistence.read-side.cassandra.keyspace = ${app.cassandra.keyspace}
elastic.url="http://localhost:9200"
akka.actor {
serialization-bindings {
# commands won't use play-json but Akka's jackson support
"com.example.app.datasource.impl.command.CommandSerializable" = jackson-json
}
}
include classpath("application.conf")
play.http.secret.key=abcdefghi12345567890
pidfile.path=/home/app/service.pid
elastic.url="http://elasticsearch:9200"
elastic.url=${?ELASTIC_URL}
cassandra.default {
username = "cassandra"
username = ${?CASSANDRA_USERNAME}
password = "cassandra"
password = ${?CASSANDRA_PASSWORD}
## list the contact points here
contact-points = ["cassandra"]
## override Lagom’s ServiceLocator-based ConfigSessionProvider
session-provider = akka.persistence.cassandra.ConfigSessionProvider
}
cassandra-journal {
contact-points = ${cassandra.default.contact-points}
session-provider = ${cassandra.default.session-provider}
authentication {
username = ${cassandra.default.username}
password = ${cassandra.default.password}
}
}
cassandra-snapshot-store {
contact-points = ${cassandra.default.contact-points}
session-provider = ${cassandra.default.session-provider}
authentication {
username = ${cassandra.default.username}
password = ${cassandra.default.password}
}
}
lagom.persistence.read-side.cassandra {
contact-points = ${cassandra.default.contact-points}
session-provider = ${cassandra.default.session-provider}
authentication {
username = ${cassandra.default.username}
password = ${cassandra.default.password}
}
}
lagom.services {
app-cron = "http://app-cron:9000"
app-datasource = "http://app-datasource:9000"
app-process = "http://app-process:9000"
test = "http://test:9000"
}
akka.management.http {
bind-hostname=${?AKKA_NAME}
hostname=${?AKKA_NAME}
port=8558
}
akka.cluster.seed-nodes = [
"akka://application@app-cron:25520",
"akka://application@app-datasource:25520",
"akka://application@app-process:25520",
"akka://application@test:25520"
]
akka {
actor {
# provider=remote is possible, but prefer cluster
provider = cluster
}
remote {
artery {
transport = tcp # See Selecting a transport below
canonical.hostname = ${?AKKA_NAME}
canonical.port = 25520
}
}
}
akka.management {
cluster.bootstrap {
contact-point-discovery {
discovery-method = akka-dns
required-contact-point-nr = 1
}
}
}
akka {
discovery {
//method = akka-dns
method = aggregate
aggregate {
discovery-methods = ["akka-dns", "config"]
}
config {
services {
kafka {
endpoints = [
{
host = "kafka"
port = 9092
}
]
}
cassandra {
endpoints = [
{
host = "cassandra"
port = 9042
}
]
}
app-cron {
endpoints = [
{
host = "app-cron"
port = 25520
}
]
}
app-datasource {
endpoints = [
{
host = "app-datasource"
port = 25520
}
]
}
app-process {
endpoints = [
{
host = "app-process"
port = 25520
}
]
}
test {
endpoints = [
{
host = "test"
port = 25520
}
]
}
}
}
}
}
I don’t quite understand what is going on where and don’t know where to go in solving it. Can someone offer suggestions??
I appreicate any help that is offered.
Thank you!!!