Issue: Unusual CPU behaviour when stopping actors

pranavkapoorr · October 5, 2018, 6:03pm

Hi everyone

When I stop parent actor which further stops all it’s children, shoots up the CPU usage to 100%.

Any help or suggestion will be highly appreciated

thanks in advance

Pranav

johanandren · October 9, 2018, 7:42am

Take a stack dump using jvisualvm or jstack and look at the stack of the threads to see what they are busy doing and you should be able to figure it out.

pranavkapoorr · October 9, 2018, 4:26pm

Hi johanandren

I did the same but unable to understand from the stack.

attaching three dumps:

first when I trigger 10 transactions
second when after 1 successful one it gets stuck
third when it timesout

can you please have a look and suggest.

many thanks

pranavkapoorr · October 9, 2018, 4:29pm

third:

2018-10-09 12:09:33
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode):

"IPS-SYSTEM-akka.actor.default-dispatcher-17" #35 prio=5 os_prio=31 tid=0x00007f8020012800 nid=0x9907 waiting on condition [0x000070000fd78000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
        at akka.dispatch.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
        at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
        - None

"IPS-SYSTEM-akka.actor.default-dispatcher-12" #30 prio=5 os_prio=31 tid=0x00007f80201ec800 nid=0x5d03 waiting on condition [0x000070000fb72000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
        at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
        - None

"IPS-SYSTEM-akka.actor.default-dispatcher-14" #32 prio=5 os_prio=31 tid=0x00007f8020b2e800 nid=0x9c03 waiting on condition [0x000070000fa6f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
        at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
        - None

"IPS-SYSTEM-akka.actor.default-dispatcher-11" #29 prio=5 os_prio=31 tid=0x00007f80208bc800 nid=0x5a03 waiting on condition [0x000070000f869000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
        at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
        - None

"IPS-SYSTEM-akka.actor.default-dispatcher-10" #28 prio=5 os_prio=31 tid=0x00007f80208b8800 nid=0xa003 waiting on condition [0x000070000f766000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
        at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
        - None

"IPS-SYSTEM-akka.actor.default-dispatcher-9" #27 prio=5 os_prio=31 tid=0x00007f80201bb800 nid=0xa103 waiting on condition [0x000070000f663000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
        at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
        - None

"IPS-SYSTEM-akka.actor.default-dispatcher-8" #26 prio=5 os_prio=31 tid=0x00007f80201bb000 nid=0xa203 waiting on condition [0x000070000f560000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
        at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
        - None

"IPS-SYSTEM-akka.actor.default-dispatcher-7" #25 prio=5 os_prio=31 tid=0x00007f801d8e0800 nid=0x4607 waiting on condition [0x000070000ec45000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
        at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
        - None

"IPS-SYSTEM-akka.actor.default-dispatcher-6" #24 prio=5 os_prio=31 tid=0x00007f80208a1800 nid=0x4107 waiting on condition [0x000070000eb42000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
        at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
        - None

"RMI TCP Connection(4)-192.168.0.20" #23 daemon prio=9 os_prio=31 tid=0x00007f801dc20800 nid=0xa303 runnable [0x000070000f45c000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
        - locked <0x0000000740899930> (a java.io.BufferedInputStream)
        at java.io.FilterInputStream.read(FilterInputStream.java:83)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$188/1900076579.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x0000000740acd010> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"JMX server connection timeout 22" #22 daemon prio=9 os_prio=31 tid=0x00007f801d85d800 nid=0xa503 in Object.wait() [0x000070000f35a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168)
        - locked <0x0000000740acdfd8> (a [I)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"RMI Scheduler(0)" #21 daemon prio=9 os_prio=31 tid=0x00007f801dc1d800 nid=0xa703 waiting on condition [0x000070000f257000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x0000000740896808> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"RMI TCP Connection(1)-192.168.0.20" #20 daemon prio=9 os_prio=31 tid=0x00007f801f926000 nid=0xa803 runnable [0x000070000f153000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
        - locked <0x00000007408d80e8> (a java.io.BufferedInputStream)
        at java.io.FilterInputStream.read(FilterInputStream.java:83)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$188/1900076579.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x0000000740acd1b0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"RMI TCP Accept-0" #19 daemon prio=9 os_prio=31 tid=0x00007f80208f1800 nid=0x1007 runnable [0x000070000f051000]
   java.lang.Thread.State: RUNNABLE
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"Attach Listener" #18 daemon prio=9 os_prio=31 tid=0x00007f801d85d000 nid=0xc07 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"DestroyJavaVM" #17 prio=5 os_prio=31 tid=0x00007f801d859000 nid=0x1703 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"IPS-SYSTEM-akka.io.pinned-dispatcher-5" #14 prio=5 os_prio=31 tid=0x00007f801fad2000 nid=0x4403 runnable [0x000070000ee4b000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
        at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
        at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x000000074030d958> (a sun.nio.ch.Util$3)
        - locked <0x000000074030d968> (a java.util.Collections$UnmodifiableSet)
        - locked <0x000000074030d908> (a sun.nio.ch.KQueueSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
        at akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.tryRun(SelectionHandler.scala:122)
        at akka.io.SelectionHandler$ChannelRegistryImpl$Task.run(SelectionHandler.scala:240)
        at akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.run(SelectionHandler.scala:155)
        at akka.util.SerializedSuspendableExecutionContext.run$1(SerializedSuspendableExecutionContext.scala:67)
        at akka.util.SerializedSuspendableExecutionContext.run(SerializedSuspendableExecutionContext.scala:71)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:38)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - <0x000000074030da18> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"IPS-SYSTEM-akka.actor.default-dispatcher-4" #12 prio=5 os_prio=31 tid=0x00007f801fabf800 nid=0x4303 waiting on condition [0x000070000ed48000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
        at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
        - None

"IPS-SYSTEM-scheduler-1" #9 prio=5 os_prio=31 tid=0x00007f801e660000 nid=0x4703 sleeping[0x000070000ea3f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:85)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:265)
        at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
        at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
        - None

"Service Thread" #8 daemon prio=9 os_prio=31 tid=0x00007f801e803000 nid=0x3d03 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C1 CompilerThread2" #7 daemon prio=9 os_prio=31 tid=0x00007f801d81b800 nid=0x4a03 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C2 CompilerThread1" #6 daemon prio=9 os_prio=31 tid=0x00007f801f024000 nid=0x4b03 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007f801e822000 nid=0x3903 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007f801f022000 nid=0x3807 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007f801d818800 nid=0x5203 in Object.wait() [0x000070000e32a000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x00000007402dce08> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:212)

   Locked ownable synchronizers:
        - None

"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007f801f014000 nid=0x3003 in Object.wait() [0x000070000e227000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        - locked <0x0000000740449178> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

   Locked ownable synchronizers:
        - None

"VM Thread" os_prio=31 tid=0x00007f801f00f000 nid=0x2e03 runnable 

"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007f801e00c000 nid=0x1a07 runnable 

"GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007f801e00c800 nid=0x1c03 runnable 

"GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007f801e00d000 nid=0x2b03 runnable 

"GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007f801e00e000 nid=0x5403 runnable 

"VM Periodic Task Thread" os_prio=31 tid=0x00007f801f018800 nid=0x4903 waiting on condition 

JNI global references: 727

------------------------------------------------------

jrudolph · October 10, 2018, 8:14am

Stack traces don’t look as if anything would be running at all. Can you use something like top -H to find out which thread goes to 100% CPU, then get a stack trace at the same time.

Then you can match the 100% CPU thread with ids in the stack trace by looking at the nid= value.

pranavkapoorr · October 10, 2018, 4:03pm

These didnt get posted
first:

2018-10-09 12:03:05
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode):

"Attach Listener" #18 daemon prio=9 os_prio=31 tid=0x00007f801d85d000 nid=0xc07 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"DestroyJavaVM" #17 prio=5 os_prio=31 tid=0x00007f801d859000 nid=0x1703 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-akka.io.pinned-dispatcher-5" #14 prio=5 os_prio=31 tid=0x00007f801fad2000 nid=0x4403 runnable [0x000070000ee4b000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x000000074030d958> (a sun.nio.ch.Util$3)
	- locked <0x000000074030d968> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000074030d908> (a sun.nio.ch.KQueueSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	at akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.tryRun(SelectionHandler.scala:122)
	at akka.io.SelectionHandler$ChannelRegistryImpl$Task.run(SelectionHandler.scala:240)
	at akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.run(SelectionHandler.scala:155)
	at akka.util.SerializedSuspendableExecutionContext.run$1(SerializedSuspendableExecutionContext.scala:67)
	at akka.util.SerializedSuspendableExecutionContext.run(SerializedSuspendableExecutionContext.scala:71)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:38)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- <0x000000074030da18> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"IPS-SYSTEM-akka.actor.default-dispatcher-4" #12 prio=5 os_prio=31 tid=0x00007f801fabf800 nid=0x4303 waiting on condition [0x000070000ed48000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
	at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-akka.actor.default-dispatcher-3" #11 prio=5 os_prio=31 tid=0x00007f801fab0800 nid=0x4103 waiting on condition [0x000070000ec45000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
	at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-akka.actor.default-dispatcher-2" #10 prio=5 os_prio=31 tid=0x00007f801faa6800 nid=0x4603 waiting on condition [0x000070000eb42000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
	at akka.dispatch.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
	at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-scheduler-1" #9 prio=5 os_prio=31 tid=0x00007f801e660000 nid=0x4703 waiting on condition [0x000070000ea3f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:85)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:265)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

"Service Thread" #8 daemon prio=9 os_prio=31 tid=0x00007f801e803000 nid=0x3d03 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C1 CompilerThread2" #7 daemon prio=9 os_prio=31 tid=0x00007f801d81b800 nid=0x4a03 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread1" #6 daemon prio=9 os_prio=31 tid=0x00007f801f024000 nid=0x4b03 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007f801e822000 nid=0x3903 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007f801f022000 nid=0x3807 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007f801d818800 nid=0x5203 in Object.wait() [0x000070000e32a000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000007402dce08> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
	- locked <0x00000007402dce08> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:212)

   Locked ownable synchronizers:
	- None

"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007f801f014000 nid=0x3003 in Object.wait() [0x000070000e227000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0000000740449178> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	- locked <0x0000000740449178> (a java.lang.ref.Reference$Lock)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

   Locked ownable synchronizers:
	- None

"VM Thread" os_prio=31 tid=0x00007f801f00f000 nid=0x2e03 runnable 

"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007f801e00c000 nid=0x1a07 runnable 

"GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007f801e00c800 nid=0x1c03 runnable 

"GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007f801e00d000 nid=0x2b03 runnable 

"GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007f801e00e000 nid=0x5403 runnable 

"VM Periodic Task Thread" os_prio=31 tid=0x00007f801f018800 nid=0x4903 waiting on condition 

JNI global references: 579

–
second:

2018-10-09 12:09:20
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode):

"IPS-SYSTEM-akka.actor.default-dispatcher-17" #35 prio=5 os_prio=31 tid=0x00007f8020012800 nid=0x9907 runnable [0x000070000fd78000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
	at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-akka.actor.default-dispatcher-15" #33 prio=5 os_prio=31 tid=0x00007f8020b2f800 nid=0x9a03 runnable [0x000070000fc75000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
	at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-akka.actor.default-dispatcher-12" #30 prio=5 os_prio=31 tid=0x00007f80201ec800 nid=0x5d03 runnable [0x000070000fb72000]
   java.lang.Thread.State: RUNNABLE
	at com.ips.altapaylink.actors.tcp.TcpClientActor.lambda$2(TcpClientActor.java:71)
	at com.ips.altapaylink.actors.tcp.TcpClientActor$$Lambda$234/263697486.apply(Unknown Source)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
	at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
	at scala.PartialFunction.applyOrElse(PartialFunction.scala:123)
	at scala.PartialFunction.applyOrElse$(PartialFunction.scala:122)
	at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
	at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
	at akka.actor.Actor.aroundReceive(Actor.scala:513)
	at akka.actor.Actor.aroundReceive$(Actor.scala:511)
	at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:132)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
	at akka.actor.ActorCell.invoke(ActorCell.scala:496)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-akka.actor.default-dispatcher-14" #32 prio=5 os_prio=31 tid=0x00007f8020b2e800 nid=0x9c03 runnable [0x000070000fa6f000]
   java.lang.Thread.State: RUNNABLE
	at sun.misc.Unsafe.unpark(Native Method)
	at akka.dispatch.forkjoin.ForkJoinPool.signalWork(ForkJoinPool.java:1955)
	at akka.dispatch.forkjoin.ForkJoinPool.externalPush(ForkJoinPool.java:1829)
	at akka.dispatch.forkjoin.ForkJoinPool.execute(ForkJoinPool.java:2955)
	at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool.execute(ForkJoinExecutorConfigurator.scala:29)
	at akka.dispatch.ExecutorServiceDelegate.execute(ThreadPoolBuilder.scala:211)
	at akka.dispatch.ExecutorServiceDelegate.execute$(ThreadPoolBuilder.scala:211)
	at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.execute(Dispatcher.scala:39)
	at akka.dispatch.Dispatcher.registerForExecution(Dispatcher.scala:115)
	at akka.dispatch.Mailbox.run(Mailbox.scala:228)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-akka.actor.default-dispatcher-13" #31 prio=5 os_prio=31 tid=0x00007f80201eb800 nid=0x5b03 runnable [0x000070000f96c000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
	at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-akka.actor.default-dispatcher-11" #29 prio=5 os_prio=31 tid=0x00007f80208bc800 nid=0x5a03 runnable [0x000070000f869000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
	at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-akka.actor.default-dispatcher-10" #28 prio=5 os_prio=31 tid=0x00007f80208b8800 nid=0xa003 runnable [0x000070000f766000]
   java.lang.Thread.State: RUNNABLE
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
	at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-akka.actor.default-dispatcher-9" #27 prio=5 os_prio=31 tid=0x00007f80201bb800 nid=0xa103 runnable [0x000070000f663000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
	at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-akka.actor.default-dispatcher-8" #26 prio=5 os_prio=31 tid=0x00007f80201bb000 nid=0xa203 runnable [0x000070000f560000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
	at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-akka.actor.default-dispatcher-7" #25 prio=5 os_prio=31 tid=0x00007f801d8e0800 nid=0x4607 runnable [0x000070000ec45000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
	at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-akka.actor.default-dispatcher-6" #24 prio=5 os_prio=31 tid=0x00007f80208a1800 nid=0x4107 runnable [0x000070000eb42000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
	at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
	- None

"RMI TCP Connection(4)-192.168.0.20" #23 daemon prio=9 os_prio=31 tid=0x00007f801dc20800 nid=0xa303 runnable [0x000070000f45c000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	- locked <0x0000000740899930> (a java.io.BufferedInputStream)
	at java.io.FilterInputStream.read(FilterInputStream.java:83)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$188/1900076579.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- <0x0000000740acd010> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"JMX server connection timeout 22" #22 daemon prio=9 os_prio=31 tid=0x00007f801d85d800 nid=0xa503 in Object.wait() [0x000070000f35a000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168)
	- locked <0x0000000740acdfd8> (a [I)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

"RMI Scheduler(0)" #21 daemon prio=9 os_prio=31 tid=0x00007f801dc1d800 nid=0xa703 waiting on condition [0x000070000f257000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000740896808> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

"RMI TCP Connection(1)-192.168.0.20" #20 daemon prio=9 os_prio=31 tid=0x00007f801f926000 nid=0xa803 runnable [0x000070000f153000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
	- locked <0x00000007408d80e8> (a java.io.BufferedInputStream)
	at java.io.FilterInputStream.read(FilterInputStream.java:83)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:550)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$188/1900076579.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- <0x0000000740acd1b0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"RMI TCP Accept-0" #19 daemon prio=9 os_prio=31 tid=0x00007f80208f1800 nid=0x1007 runnable [0x000070000f051000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
	at java.net.ServerSocket.accept(ServerSocket.java:513)
	at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:400)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:372)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

"Attach Listener" #18 daemon prio=9 os_prio=31 tid=0x00007f801d85d000 nid=0xc07 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"DestroyJavaVM" #17 prio=5 os_prio=31 tid=0x00007f801d859000 nid=0x1703 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-akka.io.pinned-dispatcher-5" #14 prio=5 os_prio=31 tid=0x00007f801fad2000 nid=0x4403 runnable [0x000070000ee4b000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x000000074030d958> (a sun.nio.ch.Util$3)
	- locked <0x000000074030d968> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000074030d908> (a sun.nio.ch.KQueueSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	at akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.tryRun(SelectionHandler.scala:122)
	at akka.io.SelectionHandler$ChannelRegistryImpl$Task.run(SelectionHandler.scala:240)
	at akka.io.SelectionHandler$ChannelRegistryImpl$$anon$3.run(SelectionHandler.scala:155)
	at akka.util.SerializedSuspendableExecutionContext.run$1(SerializedSuspendableExecutionContext.scala:67)
	at akka.util.SerializedSuspendableExecutionContext.run(SerializedSuspendableExecutionContext.scala:71)
	at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:38)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- <0x000000074030da18> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"IPS-SYSTEM-akka.actor.default-dispatcher-4" #12 prio=5 os_prio=31 tid=0x00007f801fabf800 nid=0x4303 runnable [0x000070000ed48000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000074030dbc8> (a akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
	at akka.dispatch.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

   Locked ownable synchronizers:
	- None

"IPS-SYSTEM-scheduler-1" #9 prio=5 os_prio=31 tid=0x00007f801e660000 nid=0x4703 sleeping[0x000070000ea3f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at akka.actor.LightArrayRevolverScheduler.waitNanos(LightArrayRevolverScheduler.scala:85)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:265)
	at akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

"Service Thread" #8 daemon prio=9 os_prio=31 tid=0x00007f801e803000 nid=0x3d03 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C1 CompilerThread2" #7 daemon prio=9 os_prio=31 tid=0x00007f801d81b800 nid=0x4a03 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread1" #6 daemon prio=9 os_prio=31 tid=0x00007f801f024000 nid=0x4b03 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007f801e822000 nid=0x3903 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007f801f022000 nid=0x3807 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007f801d818800 nid=0x5203 in Object.wait() [0x000070000e32a000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
	- locked <0x00000007402dce08> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:212)

   Locked ownable synchronizers:
	- None

"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007f801f014000 nid=0x3003 in Object.wait() [0x000070000e227000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	- locked <0x0000000740449178> (a java.lang.ref.Reference$Lock)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

   Locked ownable synchronizers:
	- None

"VM Thread" os_prio=31 tid=0x00007f801f00f000 nid=0x2e03 runnable 

"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007f801e00c000 nid=0x1a07 runnable 

"GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007f801e00c800 nid=0x1c03 runnable 

"GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007f801e00d000 nid=0x2b03 runnable 

"GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007f801e00e000 nid=0x5403 runnable 

"VM Periodic Task Thread" os_prio=31 tid=0x00007f801f018800 nid=0x4903 waiting on condition 

JNI global references: 681

johanandren · October 10, 2018, 4:08pm

Nothing obvious there either, what Java version and OS are you on?

pranavkapoorr · October 10, 2018, 8:47pm

I am using java 8 and tried it running on MAC OS, Windows 10 as well as Ubuntu

Topic		Replies	Views
Akka Dispatcher threads with no stacktrace Play Framework akka , akka-http , java	1	767	November 7, 2018
Akka a lot of (100+) threads with "TIMED_WAITING (parking)" Akka	4	2251	March 22, 2022
Akka dispatcher tuning cause cpu-system usage Actors akka	3	670	May 10, 2023
Akka - low CPU usage (could my code go faster?) Akka scala , configuration	5	1167	September 4, 2020
Akka Http - Performance - CPU % Akka HTTP akka , akka-http , scala	12	3283	August 6, 2020

Issue: Unusual CPU behaviour when stopping actors

Related Topics