All default-dispatcher threads run "full"

Akka Libraries Akka HTTP
1

Hi

I don’t really know how to describe the situation. After a certain load and time (around 1 h) our application “freezes”. This only happens under macOS. The resulting pattern shown in the attached screenshot builds up over time, where one thread after the other becomes like this until the freeze. When running inside a docker container based on “openjdk:8-jdk-slim” everything works fine. There I don’t see the pattern emerging.

Does maybe anybody know what could be causing the lock-up? I don’t really understand the thread dump, but maybe someone with deeper knowledge of akka-http can maybe give me a hint? In this particular scenario, we are extensively using the akka-http client to connect to an external service.

Versions:
akka: 2.5.13
akka-http: 10.1.3
Java: 1.8.0_161

Cheers,
Ivan

05
Screenshot 2018-09-05 09.38.05.png1336×1171 132 KB

(Thread dump shortened because of post limit)
DEFAULT Waiting CPU usage on sample: 0ms
java.lang.Object.wait(long) Object.java (native)
java.util.TimerThread.mainLoop() Timer.java:552
java.util.TimerThread.run() Timer.java:505

Statistics Thread-DEFAULT-1 Waiting CPU usage on sample: 0ms
sun.misc.Unsafe.park(boolean, long) Unsafe.java (native)
java.util.concurrent.locks.LockSupport.parkNanos(Object, long) LockSupport.java:215
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) AbstractQueuedSynchronizer.java:2078
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() ScheduledThreadPoolExecutor.java:1093
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() ScheduledThreadPoolExecutor.java:809
java.util.concurrent.ThreadPoolExecutor.getTask() ThreadPoolExecutor.java:1074
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) ThreadPoolExecutor.java:1134
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:624
java.lang.Thread.run() Thread.java:748

webapi-akka.actor.default-dispatcher-102 Runnable CPU usage on sample: 45ms
sun.nio.ch.FileDispatcherImpl.preClose0(FileDescriptor) FileDispatcherImpl.java (native)
sun.nio.ch.SocketDispatcher.preClose(FileDescriptor) SocketDispatcher.java:59
sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel() SocketChannelImpl.java:838
java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel() AbstractSelectableChannel.java:234
java.nio.channels.spi.AbstractInterruptibleChannel.close() AbstractInterruptibleChannel.java:115
akka.io.TcpConnection.doCloseConnection(ActorRef, Option, Tcp$ConnectionClosed) TcpConnection.scala:309
akka.io.TcpConnection.handleClose(TcpConnection$ConnectionInfo, Option, Tcp$ConnectionClosed) TcpConnection.scala:304
akka.io.TcpConnection$$anonfun$connected$1.applyOrElse(Object, Function1) TcpConnection.scala:102
scala.PartialFunction$OrElse.applyOrElse(Object, Function1) PartialFunction.scala:172
akka.actor.Actor.aroundReceive(PartialFunction, Object) Actor.scala:517
akka.actor.Actor.aroundReceive$(Actor, PartialFunction, Object) Actor.scala:515
akka.io.TcpConnection.aroundReceive(PartialFunction, Object) TcpConnection.scala:31
akka.actor.ActorCell.receiveMessage(Object) ActorCell.scala:588
akka.actor.ActorCell.invoke(Envelope) ActorCell.scala:557
akka.dispatch.Mailbox.processMailbox(int, long) Mailbox.scala:258
akka.dispatch.Mailbox.run() Mailbox.scala:225
akka.dispatch.Mailbox.exec() Mailbox.scala:235
akka.dispatch.forkjoin.ForkJoinTask.doExec() ForkJoinTask.java:260
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinTask) ForkJoinPool.java:1339
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool$WorkQueue) ForkJoinPool.java:1979
akka.dispatch.forkjoin.ForkJoinWorkerThread.run() ForkJoinWorkerThread.java:107

webapi-akka.actor.default-dispatcher-103 Runnable CPU usage on sample: 45ms
sun.nio.ch.FileDispatcherImpl.preClose0(FileDescriptor) FileDispatcherImpl.java (native)
sun.nio.ch.SocketDispatcher.preClose(FileDescriptor) SocketDispatcher.java:59
sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel() SocketChannelImpl.java:838
java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel() AbstractSelectableChannel.java:234
java.nio.channels.spi.AbstractInterruptibleChannel.close() AbstractInterruptibleChannel.java:115
akka.io.TcpConnection.doCloseConnection(ActorRef, Option, Tcp$ConnectionClosed) TcpConnection.scala:309
akka.io.TcpConnection.handleClose(TcpConnection$ConnectionInfo, Option, Tcp$ConnectionClosed) TcpConnection.scala:304
akka.io.TcpConnection$$anonfun$connected$1.applyOrElse(Object, Function1) TcpConnection.scala:102
scala.PartialFunction$OrElse.applyOrElse(Object, Function1) PartialFunction.scala:172
akka.actor.Actor.aroundReceive(PartialFunction, Object) Actor.scala:517
akka.actor.Actor.aroundReceive$(Actor, PartialFunction, Object) Actor.scala:515
akka.io.TcpConnection.aroundReceive(PartialFunction, Object) TcpConnection.scala:31
akka.actor.ActorCell.receiveMessage(Object) ActorCell.scala:588
akka.actor.ActorCell.invoke(Envelope) ActorCell.scala:557
akka.dispatch.Mailbox.processMailbox(int, long) Mailbox.scala:258
akka.dispatch.Mailbox.run() Mailbox.scala:225
akka.dispatch.Mailbox.exec() Mailbox.scala:235
akka.dispatch.forkjoin.ForkJoinTask.doExec() ForkJoinTask.java:260
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinTask) ForkJoinPool.java:1339
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool$WorkQueue) ForkJoinPool.java:1979
akka.dispatch.forkjoin.ForkJoinWorkerThread.run() ForkJoinWorkerThread.java:107

webapi-akka.actor.default-dispatcher-104 Runnable CPU usage on sample: 45ms
sun.nio.ch.FileDispatcherImpl.preClose0(FileDescriptor) FileDispatcherImpl.java (native)
sun.nio.ch.SocketDispatcher.preClose(FileDescriptor) SocketDispatcher.java:59
sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel() SocketChannelImpl.java:838
java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel() AbstractSelectableChannel.java:234
java.nio.channels.spi.AbstractInterruptibleChannel.close() AbstractInterruptibleChannel.java:115
akka.io.TcpConnection.doCloseConnection(ActorRef, Option, Tcp$ConnectionClosed) TcpConnection.scala:309
akka.io.TcpConnection.handleClose(TcpConnection$ConnectionInfo, Option, Tcp$ConnectionClosed) TcpConnection.scala:304
akka.io.TcpConnection$$anonfun$connected$1.applyOrElse(Object, Function1) TcpConnection.scala:102
scala.PartialFunction$OrElse.applyOrElse(Object, Function1) PartialFunction.scala:172
akka.actor.Actor.aroundReceive(PartialFunction, Object) Actor.scala:517
akka.actor.Actor.aroundReceive$(Actor, PartialFunction, Object) Actor.scala:515
akka.io.TcpConnection.aroundReceive(PartialFunction, Object) TcpConnection.scala:31
akka.actor.ActorCell.receiveMessage(Object) ActorCell.scala:588
akka.actor.ActorCell.invoke(Envelope) ActorCell.scala:557
akka.dispatch.Mailbox.processMailbox(int, long) Mailbox.scala:258
akka.dispatch.Mailbox.run() Mailbox.scala:225
akka.dispatch.Mailbox.exec() Mailbox.scala:235
akka.dispatch.forkjoin.ForkJoinTask.doExec() ForkJoinTask.java:260
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinTask) ForkJoinPool.java:1339
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool$WorkQueue) ForkJoinPool.java:1979
akka.dispatch.forkjoin.ForkJoinWorkerThread.run() ForkJoinWorkerThread.java:107

webapi-akka.actor.default-dispatcher-105 Runnable CPU usage on sample: 45ms
sun.nio.ch.FileDispatcherImpl.preClose0(FileDescriptor) FileDispatcherImpl.java (native)
sun.nio.ch.SocketDispatcher.preClose(FileDescriptor) SocketDispatcher.java:59
sun.nio.ch.SocketChannelImpl.implCloseSelectableChannel() SocketChannelImpl.java:838
java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel() AbstractSelectableChannel.java:234
java.nio.channels.spi.AbstractInterruptibleChannel.close() AbstractInterruptibleChannel.java:115
akka.io.TcpConnection.doCloseConnection(ActorRef, Option, Tcp$ConnectionClosed) TcpConnection.scala:309
akka.io.TcpConnection.handleClose(TcpConnection$ConnectionInfo, Option, Tcp$ConnectionClosed) TcpConnection.scala:304
akka.io.TcpConnection$$anonfun$connected$1.applyOrElse(Object, Function1) TcpConnection.scala:102
scala.PartialFunction$OrElse.applyOrElse(Object, Function1) PartialFunction.scala:172
akka.actor.Actor.aroundReceive(PartialFunction, Object) Actor.scala:517
akka.actor.Actor.aroundReceive$(Actor, PartialFunction, Object) Actor.scala:515
akka.io.TcpConnection.aroundReceive(PartialFunction, Object) TcpConnection.scala:31
akka.actor.ActorCell.receiveMessage(Object) ActorCell.scala:588
akka.actor.ActorCell.invoke(Envelope) ActorCell.scala:557
akka.dispatch.Mailbox.processMailbox(int, long) Mailbox.scala:258
akka.dispatch.Mailbox.run() Mailbox.scala:225
akka.dispatch.Mailbox.exec() Mailbox.scala:235
akka.dispatch.forkjoin.ForkJoinTask.doExec() ForkJoinTask.java:260
akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinTask) ForkJoinPool.java:1339
akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool$WorkQueue) ForkJoinPool.java:1979
akka.dispatch.forkjoin.ForkJoinWorkerThread.run() ForkJoinWorkerThread.java:107

2

Hi Ivan,

do the threads block seemingly idle or do they spin with high CPU usage?

Johannes

3

Hi Johannes

The CPU usage is constantly around 33%, so I would say with high CPU usage.

Thanks.

Ivan

4

Maybe it’s a similar issue like the one discussed here for netty:

5

Do I need to do anything special when using Akka HTTP’s singleRequest?

Also what I find strange is the difference in behavior on macOS and Linux, or is it normal?

6

We should probably treat this a bug in akka-http on macOS. Can you open up an issue at https://github.com/akka/akka-http.

There have been various issues with the macOS network stack in the past. Most people run production software on Linux so macOS issues aren’t uncovered as often.

7

Thanks, I’ve created the issue: https://github.com/akka/akka-http/issues/2201

1 Like
8

Thanks, @subotic!