Does `ask` introduce ordering issues between messages and signals in failure/stopping situations?

Akka
1

Say we two actors called parent and child and parent is watching for signals, like ChildFailed, from child.

If parent uses tell to send a message to child and the child message handling logic sends back a response and then immediately blows up due to an exception at the end of its handler logic then I believe parent is guaranteed to see the response message before it sees the ChildFailed signal.

Is this ordering (between message and signal) also guaranteed if the parent sends its original message using ask?

Why would I think ask might make a difference?

My understanding is that when using ask, the system essentially creates a temporary actor between parent and child (we see that ask fills in the ActorRef in the request rather than it being filled in by the caller). So, I’m wondering if, in the failure situation described, the response might go to this temporary actor and the signal might go directly to the parent, i.e. the message and the signal might follow different paths and the asynchronous nature of everything might mean that the temporary actor introduces a delay that results in its message ending up in the parent inbox after the directly delivered signal?

Or are things done in such a way to avoid these kinds of problems, e.g. the temporary actor, created by the ask, somehow acts as a filter/router on the parent inbox and so will never change message/signal ordering relative to what would have occurred if it had not been intermediating things?

2 Likes
2

A short answer - yes, ask can cause reordering of answer-then-termination as termination-then-answer. It requires a low probability combination of events to happen, so realistically is only reproducible in a debugger by suspending the right thread at a right time.

Explanation of the exact reproduction scenario would be quite hairy, but I’ll try nonetheless. Here is what happens, step by step.
The request processing part is something like this:

  • ctx.ask(target)(...) is just calling ctx.pipeToSelf(target.ask(...))
  • target.ask(...) is using an instance of PromiseActorRef pseudo-actor from AKKA internals as a replyTo proxy. It is just a wrapper around Scala Promise, whose ! operator completes the promise in the execution context of the target
  • ctx.pipeToSelf(future) is doing future.onComplete{x => ctx.self ! (AdaptMessage(x, fn))}(ctx.executionContext), so when the future gets done, a task#1 (to execute ctx.self !) will be submitted to the execution context of the requestor and when executed, it will submit a task#2 (to execute AdaptMessage(...) and deliver the result to the requestor) to the requestor’s execution context.
  • Lastly, actor execution context I mentioned few times, or ctx.executionContext is an ActorCell.dispatcher from the untyped AKKA, for the sake of the subject it’s just another single threaded executor.

And the actor termination part (much simpler) is something like this:

  • When an actor dies, a PostStop is dispatched, and processed by this same actor
  • Then a Terminated notification is dispatched to registered watchers as an event into their mailboxes (ultimately processed by watchers’ ctx.executionContext).

Not sure if clarifications made it any easier, but basically, if the requestor executor gets preempted exactly at the moment when it does the {x => ctx.self ! (AdaptMessage(x, fn))} bit and stays away from the CPU long enough so the died responder has posted its Terminated notification to the requestor’s inbox, then you will observe the reordering.

Realistically, because of how much more the dying responder needs to do compared to a simple redispatch, that the requestor needs to do, it seems highly unlikely that you’d be able to reproduce it easily without a help of the debugger.

With a debugger though, you can add a conditional breakpoint inside ActorContextImpl.pipeToSelf and suspend just the thread that hit the breakpoint (for a quick moment and then release). Then you will see the reordering.

See “breakpoint here” in the below code fragment (pasted from recent-ish AKKA version).

  def pipeToSelf[Value](future: Future[Value])(mapResult: Try[Value] => T): Unit = {
    future.onComplete(value => /* breakpoint here */ self.unsafeUpcast ! AdaptMessage(value, mapResult))
  }
1 Like
3

I agree. In addition I think the reordering may occur in Artery if it’s a remote watch. We have a way to avoid that for ordinary tell messages but that will not work for ask since the reply destination ActorRef of the ask is different from the watcher ActorRef.

1 Like