Retry send message every 50 milliseconds fails

Akka Libraries
1

I’m simulating in a test the conversation between three actors (A, B, C)

A --->   MessageA2B   ---> B --->   MessageB2C  ---> C

When MessageB2C is successfully arrived to C then the acknowledgement is sent back to the origin.

C ---> MessageB2C_Ack ---> B ---> MessageA2B_Ack --> A

The only peculiarity of this conversation is the message MessageB2C.
MessageB2C is sent at least every 100 ms until C does not answer with its acknowledgement.

I’ve implemented this simple conversation with scala testkit framework, but the test fail in a particular situation.

When ActorB retries to send MessageB2C more then once time, then is unable to recognize the answers from ActorC. And the answers from ActorC go to deadLetters.

  test("expectNoMessage-case: actorB retries MessageB2C every 50 milliseconds") {
    val actorA = TestProbe()
    val actorC = TestProbe()
    val actorB = system.actorOf(ActorB.props(Props(classOf[TestRefWrappingActor], actorC)), "step1-case2-primary")

    actorA.send(actorB, MessageA2B())

    actorA.expectNoMessage(100.milliseconds)

    actorC.expectMsg(MessageB2C())

    // Retries form above
    actorC.expectMsg(200.milliseconds, MessageB2C())

    // Never reach this point with 100 ms frequency
    actorC.expectMsg(200.milliseconds, MessageB2C())

    actorA.expectNoMessage(100.milliseconds)

    actorC.reply(MessageB2C_Ack())

    // Never reach this point with MessageB2C 50 ms frequency
    actorA.expectMsg(MessageA2B_Ack())
  }

This is the ActorB code:

class ActorB(actorCProps: Props) extends Actor {
  import ActorB._
  import context.dispatcher

  val log = Logging(context.system, this)

  val actorC = context.actorOf(actorCProps)

  def receive = {
    case r:MessageB2C_Ack => {
      log.info("ActorB - secondary received UNHANDLED message: MessageB2C_Ack")
    }
    case r:MessageA2B => {
      val client = context.sender()
      implicit val timeout = Timeout(100.milliseconds)
      log.info("ActorB received message MessageA2B from client " + client)
      implicit val scheduler=context.system.scheduler
      val p = MessageB2C()

      RetrySupport.retry(() => {
        log.info("ActorB - sent message MessageB2C to ActorC " + actorC)
        Patterns.ask(actorC, p, 50.millisecond)
      }, 10, 50.millisecond)
      .onSuccess({
        case p: MessageB2C_Ack => {
          log.info("ActorB - Received MessageB2C_Ack so now sending an MessageA2B_Ack to client " + client)
          client ! MessageA2B_Ack()
        }
      })

    }
    case r => {
      log.info("ActorB - received UNHANDLED message " + r)
    }
  }
}
2

Something that might not be obvious about using Patterns.ask, is that it spawns a temporary actor that only exists to receive the reply and complete the Future returned from ask. That temporary actor is used as the sender of the MessageB2C message send to actor C. That means that when C replies, it is not to B, but rather to the temporary actor created by ask. In each retry, that will be a new actor, and when the ask times out, the temporary actor will be stopped.

Then, when you use expectNoMessage, the test will have to wait for the entire duration that you pass to it (100ms) before proceeding. That means that by the time you call actorC.reply(MessageB2C_Ack()), the temporary sender of the previous message will have timed out and stopped, and there should be a new retried message in actor C’s mailbox. That’s why the reply goes to dead letters. I believe you will be able to solve this by adding another actorC.expectMsg(200.milliseconds, MessageB2C()) in between actorA.expectNoMessage(100.milliseconds) and actorC.reply(MessageB2C_Ack()).

In general, using an ask from inside an actor like this is often discouraged in favor of directly modelling the request and response flow directly in the sending actor. This is often simpler and less “magical”, with fewer surprises like this one. This article explains some of the tradeoffs pretty well https://medium.com/@yuriigorbylov/akka-ask-antipattern-8361e9698b20

4

Thanks Tim for the interesting answer, this make pretty clear why my ActorB implementation wasn’t able to successfully complete the test. Looking around for an alternative, I’m trying the Error Kernel pattern.

5

So for me Error Kernel pattern was the right way to choose. I created a child actor of ActorB which try multiple message sending using a scheduler. In case of error the child will be restarted and only when the acknowledge has received the child notify the parent and stops. What do you think about?