Actor processes messages way too late, dispatcher configuration problem?

Akka Libraries
1

Hi

We use akka actors with persistence (postgresql database connected with akka-persistence-jdbc version 3.5.3) and cluster version 2.6.9 with scala version 2.13.

There are two actors A and B which live on the same node. I send messages from actor A to actor B and log:

  • right before the tell() from A to B happens
  • on receiving the message in actor B

Actor A is dispatched by the default-dispatcher and extends AbstractActor.
Actor B is dispatched by a custom blocking-persistence-dispatcher and extends AbstractPersistentActor. This actor persists the received message (but first I log the message).

After a while (not so long), the messages are sent in time as expected but actor B processes them way too late (9 seconds later). I think this problem could be related to a problem with the dispatchers.

Here is the whole dispatcher configuration setup:

# Applied to actors doing blocking input/output operations.
  blocking-io-dispatcher {
    type = Dispatcher
    executor = "thread-pool-executor"
    thread-pool-executor {
      fixed-pool-size = 32
    }
    # Throughput defines the maximum number of messages to be
    # processed per actor before the thread jumps to the next actor.
    # Set to 1 for as fair as possible.
    throughput = 1
  }

  # Applied to persistent actors
  blocking-persistence-dispatcher {
    type = Dispatcher
    executor = "thread-pool-executor"
    thread-pool-executor {
      fixed-pool-size = 32
    }
    # Throughput defines the maximum number of messages to be
    # processed per actor before the thread jumps to the next actor.
    # Set to 1 for as fair as possible.
    throughput = 1
  }

and here is the persistence configuration (with *** instead of sensitive information):

slick {
  profile = "slick.jdbc.PostgresProfile$"
  db {
    url = "jdbc:postgresql://localhost:5432/***"
    user = ***
    password = ***
    driver = "org.postgresql.Driver"

    // hikariCP
    numThreads = 4
    maxConnections = 4 // limit maxConnections to numThreads to avoid deadlocks
  }
}
  1. is this a sensible configuration setup if run on a raspberry pi compute module 3?
  2. how can I determine whether it’s the dispatcher’s fault?
  3. how can I fix this issue?

I created gc logs and analyzed them with gceasy.io and everything is fine. Pausing the application cannot be the root of the problem (max pause time is 100 ms).

Thanks!

2

Not sure it will help with the problem, but should shouldn’t need to use a dedicated dispatcher for akka-persistence-jdbc. JDCB is indeed blocking, but that is handled internally in the plugin.

3

@patriknw thanks a lot for your input. The reason why I use a dedicated dispatcher is because of snapshot replays. When I start the application and there is a persistent actor with huge snapshot to recover from, it blocks the thread for too long. This is an issue when having many persistent actors with huge snapshots.

Anyway, I removed the dedicated dispatcher to see whether it solves the problem. It didn’t.

I will replace the persistence journal and snapshot store with Null-objects which just ignore the message and return success to see (and potentially exclude) the relation to persistence. Maybe there’s an other reason why message dispatching is delayed so much.

Is there a way to prove it’s not the dispatcher’s fault?
Is there a way to monitor dispatchers to detect whether there’s a resource problem and thus actors cannot be executed in a timely manner?

If any debug information helps, please let me know what I should provide.