I’ve observed that the memory usage of the Akka Cluster nodes continuously increases, which eventually leads to pod restarts after a certain period. I can clearly see this trend while monitoring the memory usage. I’ve attached a visual representation of this issue below.
To investigate further, I’ve activated Cinnamon on the Akka cluster and started tracking the mailboxes of the system actors. Upon analyzing the data, I identified that mailboxes of actors like ActorGraphInterpreter and TcpOutgoingConnection are growing continuously. I’ve also attached the relevant visuals below.
I am trying to solve the memory issue and would like to know if I am looking at the right place. If this is indeed the right area, how can I prevent the growth of these actor mailboxes? Since these actors are system actors, I cannot intervene directly.
Any insights or suggestions would be greatly appreciated.
It would be interesting if you can attach a debugger or do a heap dump, to inspect exactly what all those messages in the inboxes are and see if that sheds a light on it.
For ActorGraphInterpreter it could mean that you are continuously spawning new streams in your application that never complete, so stay resident. For TcpOutgoingConnection connection I’d expect that if it is not in use it would hit an idle timeout and tear itself down. Neither should end up with a continuously growing mailbox as they do back pressure rather than accept a fire and forget type of protocol.
The memory graph shared looks like it could be normal heap usage with GC being able to release a lot of heap periodically and not an obvious memory leak to me. It could still be a memory leak but for a memory leak I’d generally expect continuously growing heap usage until the JVM fails with an out of memory error, or repeated GC dips that each time manages to collect less memory.
I"m not sure of that. My initial suspicion is along the lines of what @johanandren suggests: the presence of so many interpreters suggests that OP might be continually spawning new streams/materializers rather than reusing them.
