There is a UserEntity, which depends on another two entities, says ItemEntity and FoodEntity. When UserEntity receives a modify command, it fisrtly turns into a state in transaction, which means it will not accept any modify command untill it receives response from ItemEntity and FoodEntity and then make the state out of transaction.
def applyCommand(cmd: Command)(implicit setup: Setup): UserEffect = {
cmd match {
case req: ModifyRequest =>
UserProgram.onModifyRequest(this, req)
case res: ModifyResult =>
UserProgram.onModifyResult(this, res)
case _ =>
Effect.unhandled
}
}
def onModifyRequest(state: UserState, req: ModifyRequest)(implicit setup: Setup): UserEffect = {
if (state.isInTransaction) {
logError("reenter modify assets")
Effect.unhandled.thenReply(req.replyTo)(_ => ModifyRsp(RetCode.Busy.id))
} else {
val foodRef = setup.appCtx.clusterSharding.entityRefFor(FoodEntity.TypeKey, setup.userId.toString)
val itemRef = setup.appCtx.clusterSharding.entityRefFor(ItemEntity.TypeKey, setup.userId.toString)
val txnId = state.getTransactionId
val foodFuture = akka.pattern.retry(
{ () =>
foodRef.ask(FoodEntity.Modify(txnId, req, _))
},
60,
5.seconds)
val itemFuture = akka.pattern.retry(
{
() => itemRef.ask(ItemEntity.Modify(txnId, req, _))
},
60,
5.seconds)
val resultsFuture = for {
foodRes <- foodFuture
itemRes <- itemFuture
} yield {
// send another cmd to turn the state out of transaction
selfRef ! ModifyResult(retCode, req.replyTo)
}
// turn state into transaction
Effect.persist(UserEvent.TransactionBegan())
}
}
def onModifyResult(state: UserState, res: ModifyResult)(implicit setup: Setup): UserEffect = {
Effect
.persist(UserEvent.TransactionEnded())
.thenReply(res.replyTo)(_ => ModifyRsp(res.ret.id))
}
Since the interaction with ItemEntity and FoodEntity is asynchronous, one reasonable way to send the result to the ‘aggregator’ UserEntity is by generating a inner command. However, if the UserEntity can not receive that inner command due to some egde cases like send timeout or process crash or rebalance, etc. it will cause disaster since all the subsequent modify command will be refused. So here comes the question, how to ensure the reliable delivery of the command to the UserEntity?
If the UserEntity, ItemEntity and FoodEntity have strict consistent requirements, then they should not be three different entities.
In DDD terms, they would form an Aggregate and define one single consistency boundary. Translated to Akka, it means that they are one single Entity.
But from their names, they don’t seem to belong together. So maybe you should revisit the consistent requirements. It’s hard to infer from the example as the commands are just ModifyRequest.
Instead, you need to build a Saga that will let events from one entity be transformed to commands to another entity. Try to get familiar with how a Choreography Sagas works. Try first to understand how your system would benefit from a Choreography Saga. Once you have a good understand of it, have a look at Akka Projections. It will give you the missing functionality to build a choreography saga.
Hi @octonato .
I agreed much with that we should not try to keep strong consistency with more than one single Entity. In fact, this is legacy code in our project and its design might not reasonable. The data of ItemEntity and FoodEntity always seem to change at the same time, but we have to manage high complexity to keep consistency between them. I personally believe that microservice concept is somehow overused in projects nowadays. Especially in the event sourcing with cluster sharding pattern based on Akka, splitting domain appropriately seems more important than in pure crud development pattern, just like our case, the ItemEntity and FoodEntity is in the same service, but it brings the problem of distributed consistency.
Suppose we can only live with this historical debt, as you’ve said, we should adopt a distributed transaction solution like Saga, since the ItemEntity and FoodEntity probably are remote to UserEntity.
Btw, i am a new fans of Akka, but i found there exists few learning materials about builting Saga pattern. May you give me some instructions？
Learned a lot from your answer. Appreciate it!
Just put aside these two troublemakers(ItemEntity and FoodEntity), let’s say there is only one entity, i.e. UserEntity. The question degrade into this one: how to make sure the subsequent inner command to be delivered to local entity(UserEntity) to turn the state into next step?
The only reason to send a command to self, IMO, is to delay some processing. But why you want to delay? Are you setting the entity into some protected state and want to unlock?
I would avoid setting the UserEntity in a transactional state because in case of failures it will stay locked.
I think the path you should look here is to build a saga (using Akka Projection) that listen to events from UserEntity and propagate changes to FoodEntity and ItemEntity. If you want, you can set your User into some locked state and then listen to events from FoodEntity and ItemEntity to unlock your UserEntity.
If you use Akka Projections to do it, you have at-least-once guarantee. The projection will receive the event, you transform to a command to the target entity and sent it. On ack, the projection will save the offset. It may happen that the ack times out. In which case, the event will be redelivered. So you must ensure that each target entity are idempotent.
If you build a saga as such, you will have delivery guarantees (at-least-once). It will take longer then currently implemented, but it will give you delivery guarantees.
Yes you’re right, there is no point in sending a command to self which doesn’t interact with remote systems. Sorry, i’ve got wrong with the case, which is actually dependent on a response of remote call. UserEntity A send AddFriendApply to UserEntity B, after that UserEntity A turns into a transactional state in which will block any requests that updates friend data. When UserEntity A recieves confirmed reply from UserEntity, it exits the transactional state. In fact, it’s also a distributed consistency problem. However, we try to solve it in a simple and crude way:
Before UserEntity A send AddFriendApply request, it persist an event which is used to turn itself into transactional state by event handler.
And then start a retry process to make sure the command is finally recieved by UserEntity B and send replies back.
In the recovery hook, the same retry process as step two is also launched.
In a word, the system do its best effort to retry to achieve at-least-once delivery without using SAGA which brings in heavier burden to our code and system.
In the case where exists FoodEntity and ItemEntity, we will review the design and figure out whether we can maintain only one entity to avoid the distributed consistency problem. But if the answer is no, we will check out the SAGA solution.
I took a quick look at the document, but i found that there is no direct user guide or demo about building SAGA based on Akka Event source and Projection. May you give me some reference just like this(https://docs.axoniq.io/reference-guide/axon-framework/sagas/implementation)?
