Pool Slot was Shut Down

Hello,

I have a four node cluster running with K8 in Minikube locally, with sharding. Everything is functioning as normal. I use K8 API with AKAK Management for cluster formation and discovery. However, I see a lot of output stream error with pool shut down message in the log as shown below. Is there anything I need to specifically look for.

SLF4J: A number (1) of logging calls during the initialization phase have been intercepted and are
SLF4J: now being replayed. These are subject to the filtering rules of the underlying logging system.
SLF4J: See also http://www.slf4j.org/codes.html#replay
[2020-05-07 14:15:12,854] [INFO] [akka.event.slf4j.Slf4jLogger] [] [CartCluster-akka.actor.default-dispatcher-3] - Slf4jLogger started {}
[2020-05-07 14:15:27,861] [INFO] [com.example.Bootstrap$RootBehavior$] [] [CartCluster-akka.actor.default-dispatcher-3] - starting node with roles: akka.cluster.typed.internal.AdapterClusterImpl@53f4616c.selfMember.roles {akkaAddress=akka://CartCluster@172.18.0.8:2552, akkaSource=akka://CartCluster/user, sourceActorSystem=CartCluster}
[2020-05-07 14:15:38,962] [ERROR] [com.example.Bootstrap$RootBehavior$] [] [CartCluster-akka.actor.default-dispatcher-3] - AKKA management and cluster bootstrap started for K8 {akkaAddress=akka://CartCluster@172.18.0.8:2552, akkaSource=akka://CartCluster/user, sourceActorSystem=CartCluster}
[2020-05-07 14:15:55,358] [ERROR] [akka.management.cluster.bootstrap.internal.HttpContactPointBootstrap] [] [CartCluster-akka.actor.default-dispatcher-22] - Overdue of probing-failure-timeout, stop probing, signaling that it's failed {akkaAddress=akka://CartCluster@172.18.0.8:2552, sourceThread=CartCluster-akka.actor.default-dispatcher-21, akkaSource=akka://CartCluster@172.18.0.8:2552/system/bootstrapCoordinator/contactPointProbe-172-18-0-8.default.pod.cluster.local-8558, sourceActorSystem=CartCluster, akkaTimestamp=14:15:55.262UTC}
[2020-05-07 14:16:04,866] [ERROR] [com.example.ClusterListener$] [] [CartCluster-akka.actor.default-dispatcher-23] - Member is Up: akka://CartCluster@172.18.0.7:2552 {akkaAddress=akka://CartCluster@172.18.0.8:2552, akkaSource=akka://CartCluster/user/ClusterListener, sourceActorSystem=CartCluster}
[2020-05-07 14:16:07,552] [ERROR] [com.example.ClusterListener$] [] [CartCluster-akka.actor.default-dispatcher-3] - Member is Up: akka://CartCluster@172.18.0.8:2552 {akkaAddress=akka://CartCluster@172.18.0.8:2552, akkaSource=akka://CartCluster/user/ClusterListener, sourceActorSystem=CartCluster}
[2020-05-07 14:16:10,655] [ERROR] [com.example.ClusterListener$] [] [CartCluster-akka.actor.default-dispatcher-12] - Member is Up: akka://CartCluster@172.18.0.9:2552 {akkaAddress=akka://CartCluster@172.18.0.8:2552, akkaSource=akka://CartCluster/user/ClusterListener, sourceActorSystem=CartCluster}
[2020-05-07 14:16:12,051] [ERROR] [com.example.ClusterListener$] [] [CartCluster-akka.actor.default-dispatcher-22] - Member is Up: akka://CartCluster@172.18.0.10:2552 {akkaAddress=akka://CartCluster@172.18.0.8:2552, akkaSource=akka://CartCluster/user/ClusterListener, sourceActorSystem=CartCluster}
[2020-05-07 14:16:28,586] [ERROR] [akka.actor.ActorSystemImpl] [] [CartCluster-akka.actor.default-dispatcher-14] - Outgoing request stream error {akkaAddress=akka://CartCluster@172.18.0.8:2552, sourceThread=CartCluster-akka.actor.default-dispatcher-23, akkaSource=akka.actor.ActorSystemImpl(CartCluster), sourceActorSystem=CartCluster, akkaTimestamp=14:16:28.580UTC}
akka.http.impl.engine.client.pool.NewHostConnectionPool$HostConnectionPoolStage$$anon$1$Slot$$anon$2: Pool slot was shut down
[2020-05-07 14:16:28,660] [ERROR] [akka.actor.ActorSystemImpl] [] [CartCluster-akka.actor.default-dispatcher-12] - Outgoing request stream error {akkaAddress=akka://CartCluster@172.18.0.8:2552, sourceThread=CartCluster-akka.actor.default-dispatcher-14, akkaSource=akka.actor.ActorSystemImpl(CartCluster), sourceActorSystem=CartCluster, akkaTimestamp=14:16:28.660UTC}
akka.http.impl.engine.client.pool.NewHostConnectionPool$HostConnectionPoolStage$$anon$1$Slot$$anon$2: Pool slot was shut down
[2020-05-07 14:16:28,660] [ERROR] [akka.actor.ActorSystemImpl] [] [CartCluster-akka.actor.default-dispatcher-12] - Outgoing request stream error {akkaAddress=akka://CartCluster@172.18.0.8:2552, sourceThread=CartCluster-akka.actor.default-dispatcher-23, akkaSource=akka.actor.ActorSystemImpl(CartCluster), sourceActorSystem=CartCluster, akkaTimestamp=14:16:28.660UTC}
akka.http.impl.engine.client.pool.NewHostConnectionPool$HostConnectionPoolStage$$anon$1$Slot$$anon$2: Pool slot was shut down
[2020-05-07 14:16:28,661] [ERROR] [akka.actor.ActorSystemImpl] [] [CartCluster-akka.actor.default-dispatcher-14] - Outgoing request stream error {akkaAddress=akka://CartCluster@172.18.0.8:2552, sourceThread=CartCluster-akka.actor.default-dispatcher-23, akkaSource=akka.actor.ActorSystemImpl(CartCluster), sourceActorSystem=CartCluster, akkaTimestamp=14:16:28.661UTC}
akka.http.impl.engine.client.pool.NewHostConnectionPool$HostConnectionPoolStage$$anon$1$Slot$$anon$2: Pool slot was shut down
[2020-05-07 14:16:28,670] [ERROR] [akka.actor.ActorSystemImpl] [] [CartCluster-akka.actor.default-dispatcher-23] - Outgoing request stream error {akkaAddress=akka://CartCluster@172.18.0.8:2552, sourceThread=CartCluster-akka.actor.default-dispatcher-14, akkaSource=akka.actor.ActorSystemImpl(CartCluster), sourceActorSystem=CartCluster, akkaTimestamp=14:16:28.669UTC}
akka.http.impl.engine.client.pool.NewHostConnectionPool$HostConnectionPoolStage$$anon$1$Slot$$anon$2: Pool slot was shut down
[2020-05-07 14:16:28,670] [ERROR] [akka.actor.ActorSystemImpl] [] [CartCluster-akka.actor.default-dispatcher-23] - Outgoing request stream error {akkaAddress=akka://CartCluster@172.18.0.8:2552, sourceThread=CartCluster-akka.actor.default-dispatcher-14, akkaSource=akka.actor.ActorSystemImpl(CartCluster), sourceActorSystem=CartCluster, akkaTimestamp=14:16:28.670UTC}
akka.http.impl.engine.client.pool.NewHostConnectionPool$HostConnectionPoolStage$$anon$1$Slot$$anon$2: Pool slot was shut down

Thank you

Hi @alan.brewster73,

which version of akka-http and akka-management do you use? If you do not depend on akka-http directly you might use a slightly old version of akka-http pulled in by akka-management. In that case, you should add an explicit dependency to akka-http 10.1.12 which is less chatty when it comes to those messages.

This one seems related: https://github.com/akka/akka-http/issues/3041

Johannes