Difficulty managing several (nested) futures with multiple "asks" to actors

Akka Libraries
,
1

I have recently begun learning Akka, and I am trying to convert a consensus protocol that I wrote in Java to Akka typed actors in Scala. Most of what I have seen is incredibly concise and elegant, but I am finding that certain operations that need to reach out to other actors result in implementations that look anything but concise or elegant. That makes me think that I am doing it wrong.

Here is an example scenario that I would like to do in a functional and idiomatic way with typed actors. To keep actors from being coupled, I made the decision to not keep actor refs in the actor. Instead, I am using service keys to get actor refs when I need them. When I do that, now we have one level of futures to map. Next, I take that actor ref and I ask the actor for a piece of information. Now we are at two levels of futures. Since I have that information, now I will need to use the information in a message I am sending to yet another actor. Enter nested future number 3! Now, I can finally send the information since I have that actor’s ref, but wouldn’t you know it – this is future number 4. You may be thinking that there’s no real reason why I would have to wait to get the second actor ref, if I know that I am going to need it, so I could use for comprehension to do that, right? Well, if that’s not even a viable answer, it is because I am also trying to beef up my Scala skills in the meantime.

Either way, how do I handle this as elegantly as possible? I have looked through the online documentation, and I have purchased Akka In Action, but I have not seen this use case.

Here is an example of a method in one of my actors that does some nesting:

  private def processFinalizeMessage(iterationNumber: Int,
                                     localPlayerId: String,
                                     leaderId: String,
                                     finalizeReceipts: Seq[String],
                                     finalizeSigned: FinalizeSigned,
                                     receptionist: ActorRef[Receptionist.Command],
                                     timer: Cancellable,
                                     context: ActorContext[IterationCommand])
                                    (implicit timeout: Timeout, ec: ExecutionContext, scheduler: Scheduler): Behavior[IterationCommand] = {
    val finalizeMsg = finalizeSigned.finalizeMsg
    if (finalizeMsg.iteration != iterationNumber) {
      Behaviors.same
    } else {
      val finalizeReceipt = finalizeMsg.playerId
      val playerServiceRefFuture: Future[Receptionist.Listing] =
        receptionist ? Receptionist.Find(PlayerActor.playerServiceKey)

      Behaviors.receiveMessage {
        case _: IterationCommand =>
          Behaviors.same
        case newBehavior: Behavior[IterationCommand] @unchecked =>
          newBehavior
      }

      playerServiceRefFuture.flatMap { listing =>
        listing.serviceInstances(PlayerActor.playerServiceKey).headOption match {
          case Some(playerActorRef) =>
            val publicKeyFuture = playerActorRef.ask(replyTo => PlayerActor.GetPlayerPublicKey(finalizeReceipt, replyTo))
            publicKeyFuture.map {
              case Some(publicKey) =>
                if (CryptoUtil.verifySignature(MessageUtils.toBytes(finalizeMsg), finalizeSigned.signature, publicKey, MESSAGE_DIGEST_ALGORITHM)) {
                  context.self ! iterationBehavior(iterationNumber, localPlayerId, leaderId, finalizeReceipts :+ finalizeReceipt, receptionist, timer, context)
                } else {
                  val errorMessage = s"Signature verification failed for finalize receipt from player $finalizeReceipt"
                  context.log.warn(errorMessage)
                  context.self ! Behaviors.same
                }
              case None =>
                val errorMessage = s"Could not find player actor for player ID $finalizeReceipt so the finalize receipt could not be verified."
                context.log.warn(errorMessage)
                context.self ! Behaviors.same
            }.recover {
              case ex =>
                val errorMessage = s"Error occurred while processing finalize message: ${ex.getMessage}"
                context.log.error(errorMessage)
                context.self ! Behaviors.same
            }
          case None =>
            val errorMessage = s"Could not find player actor reference to verify the finalize message."
            context.log.warn(errorMessage)
            context.self ! Behaviors.same
        }
      }
    }
  }

I know that this isn’t even fully compiling code yet, but I am trying to figure some of this stuff out, so please be kind!

2

To keep actors from being coupled, I made the decision to not keep actor refs in the actor.

That’s pretty … non-idiomatic. (So is using Futures with an Actor, you can’t really do that without losing the Actor context.) A core principle of Akka is that each Actor needs to own its own state. A part of this is that the actor should be able to resolve messages in a non-blocking way. If the Akka has to rely on an asynchronous call to an external entity to even send a message … that’s going to cause all kinds of issues.

The best analogy I can think of is saying “to keep Java objects from tightly coupled, we instead keep a physical memory address of the referenced object. But that means we need to make a JNI call for each lookup. And, of course, we need to build or own type system to resolve the reference”. It doesn’t really prevent coupling and it’s its extremely painful.

1 Like
3

Ok. Thanks for letting me know. So, do you think you can offer some example, or something, to show me how to do it better? I enjoy the feedback, but I can’t do much if I don’t have a way to learn to do it better.

4

Fundamentally, it just seems like you are processing messages. Rather than using the ask pattern just:

  1. When you Process Finalize Message message., check to see if you already have the public key for that player (which you keep in a HashMap in the actor context). If so, just do your logic. If not send a “GetPlayerPublicKey” and stash the current message.
  2. Conversely, when a Player actor gets a GetPlayerPublic key message, it can just reply with the public key.
  3. When this actor gets a "PlayerPublicKeyResponse’ message, it can add the key to its hash map of known public keys and then unstash the messages.

Maybe I’m misunderstanding your business logic. But that’s gist is the same: rather than architecting around Futures, design a series of messages that forms the back and forth between the actors.

5

To give you more context, I’ll explain a bit about what I’m doing for this actor. I have an implementation of a blockchain consensus protocol that I wrote in Java in the form of traditional services, and I thought that it would be the perfect type of thing to implement with Akka so that I could learn the actor model. During the course of a consensus iteration, other “players” might send a finalize message that is signed with their public key. So, then, I have to get the public key from the player that sent the finalize message to verify the signature in order to know if it’s a valid message. If it’s valid, then I record a receipt, of sorts, of the finalize message for the current iteration.

When I said that I didn’t want to hold onto references to other actors, I did not mean it in the sense of caching. My point was more about not instantiating the actors with actor refs with which the actor needs to interact in order to avoid coupling actors together. It gets more complicated if there are actors that are short-lived. Either way, I agree that caching is beneficial. In this case, I feel that caching is an optimization, but it does not inherently or drastically change how I do things. If I do not have this actor ref cached, I will have to ask the receptionist for the ref.

My uncertainty pertains more to the overall code, where I am using multiple futures, where they end up getting nested because of the order in which I need to get, and use, the information. If you have some advice on using multiple futures (in general) where I need to get current information from actors, I would greatly appreciate the information. Note that my explanation was just my description of how I see things coming mostly from the Java world. Since the actor model is a new paradigm for me, I completely accept that it will be helpful to change my perspective, and I’ve outlined things a bit so that you can nudge me in different directions to help me adjust, as necessary. Thanks again for your time!

6

More regarding caching, as a bit of an aside to my main future-management question… I had thought about creating some common code that handles the asking and receipt of an actor ref from the receptionist. Perhaps I can create a trait that uses a service key to look up an actor ref from a cache that loads up the reference if it is not present, and all of my actors can extend that. Is that something that people typically do? Are there other libraries that facilitate this kind of thing?

7

Please forgive the flood of messages, but I realized that I overlooked one bit of advice that you provided in your last message. You said that I could ask the PlayerActor for the public key of a player, and stash the message. I want to make sure that I understand your meaning. Are you suggesting that I fire off this message, but not set up a future to handle it right there? Instead, use this stashed message as a way to correlate the response handling, and thereby avoiding the future nesting that my very novice code snippet is doing? To put it another way, rather than coding the workflow inline, I would devise some kind of “message flow choreography” to handle these workflows? I have a pretty extensive background in EIP implementations and usage (e.g., Apache Camel), so this is an idea that I can understand and I wouldn’t be opposed to seeing how badly I could mangle it in Scala!

The other item was that the PlayerActor can “just reply with the player public key” when it receives the message that requests that. I am curious what prompted this comment, since I am not sure what other way I could handle this. The PlayerActor does simply return the PublicKey instance for a playerId, if it has it. So, I think I must be misunderstanding what you mean, or I’ve done something wrong that I’m not quite aware of. Again, thanks for your patience and advice.

8

Yes, “message flow choreography” is a good way of describing what I was trying to say.

I mean, you can use the ask pattern and use Futures, but behind the scenes what that pattern is doing is sending a message back to the actor anyway. (It has to, so that the actor has the context and the threading model is preserved). In my personal opinion (and I’m open to disagreement, I’ve been away from Akka a long time in my day to day), it’s always better to favor “message choreography”.

An actor fundamentally needs to make its decisions on only its internal state (including its current Behavior) and the message it has received (including the message type) . If using the ask pattern helps you model the interactions more easily, that’s fine. But if the ask pattern is making it more complicated because it’s got too much back and forth (as it seems like you are implying) then trying to figure out the message choreography “manually” might help.

For me, at least, I find it easier to think about the sequence of events from the perspective of Behaviors and messages. I avoid Futures completely, unless I have no other choice.

If I only have to deal with one Finalize message at a tiime I start in a “normal” behavior. Doing whatever I do. But when I receive a “finalize” message I see that if I have the information that I need to process it. If I do, obviously that’s easy. But if I don’t have a needed ActorRef I send a (fire and forget “tell”) message to the Receptionist. and switch to a “WaitingForReceptionist” Behavior. I then either stash the original message or update my state in some way to store that partially completed “transaction”.

While in WaitingForReceptionist I stash anything I shouldn’t process while waiting, process anything I can, and wait for that Listing message back from the Receptionist. When I get that message back I can attempt to process it again. If I can, I do and switch back to “normal” Behavior.

But, if I can’t process that original finalization because I still don’t have the public key, then I go back into a WaitingForPublicKey Behavior. Which operates similarly: it handles what it can, stashes what it can’t, and generally waits for the reply to that message.

I personally find this “state machine” kind of message processing easier clearer and more idiomatic than a bunch of Futures/pipeTos, and implied state. Especially in cases where you need to be explicit about what can and can’t happen while in the middle of handling one of these “saga like” activities. i.e. when I have to decide what my actor can and can’t do while processing a finalization.

1 Like
9

Thank you for taking the time to write that thorough reply, and it was very helpful. I enjoy learning new paradigms that offer elegant solutions to various problem sets, and I tend to jump in the deep end too fast, so that the immersion forces me to learn as much, and as quickly, as possible.

I’d be interested in what you think about using child actors for tasks/workflows that involve the coordination of several (maybe sequential) steps. If multiple messages of the same type need to be processed, and their processing time might overlap, this would provide the necessary correlation for free, and allow the parent actor to kick off processing anytime that type of message is received. To avoid nesting the futures from asking for information, the child actor would tell itself the result, which would prompt it to take the action for the next step in the flow, but simplify the Future processing. And I am not asking about this as a replacement for what you just suggested, but just another approach when it could be useful. I think that your advice would fit in pretty well with the child actor approach, too. Thoughts?

1 Like
10

I hope david follows up because I’ve really enjoyed this thread, but I wanted to share my thoughts in the meantime. I’ve been tinkering with Akka 2.6’s typed Behaviors DSL for the last year or so but I don’t necessarily know what the best practices are, though I’ve tried to pick things up from the docs and I’ve learned repeatedly not to use ActorContexts in Futures :sweat_smile:

using child actors for tasks/workflows that involve the coordination of several (maybe sequential) steps

This is part of how I’ve started thinking of it. Each of my actors is a state machine, and that often means having an initializing state. I’ve started liking that because it models the flow better anyway, but I always disliked having multiple init steps because it cluttered my business logic.

In my use case, actors that want to write Obsidian notes (Markdown stored in a “vault” folder) needed to request an exclusive NoteRef via the VaultKeeper, found via Receptionist. So that’s 3 init states

  • waitingForReceptionist
  • waitingForVaultKeeperListing
  • waitingForNoteRef.

Actors that want to write notes just care about getting a NoteRef, so I refactored this into a NoteRefFetcher. Its constructor takes the note name and replyTo ActoRef and encapsulates all that logic, so my actors can have a SINGLE init state just waiting for the NoteRef. Then NoteRefFetcher just uses Behaviors.ignore or Behaviors.stopped when it’s done.

I’m new to using the actor model so I tried using traits and inheritance at first, but I read somewhere that this can be good though I can’t cite that source unfortunately. I can just cite my (admittedly possibly mistaken) personal experience that it’s been a huge relief. If I find the citation I’ll post back.

To avoid nesting the futures from asking for information, the child actor would tell itself the result

My only use-case for nested futures so far has been using Akka HTTP where I get a Future[HttpResponse] which lets me unmarshal and get another future with the payload I want. I’ve learned that anytime I get a Future, I should immediately pipeToSelf and use states to manage things. I always use the “fire and forget” tell/! and then just provide a new Behavior for the new waiting state if necessary, avoiding Futures entirely if I can.

Having all the states seemed onerous at first but now I feel like I was just trying (but failing) to avoid it before and now I’m more accurately modeling (and encapsulating) complexity.

1 Like