Does Akka Persistence provide delivery guarantees for Event Storage?

Akka Libraries Persistence / Event Sourcing
1

Does Akka Persistence provide delivery guarantees, in the meaning to guarantee that an event is actually stored in the event storage (eg Cassandra)? Otherwise, if there is no guarantee that an event was successfully persisted, this would make the whole Persistence model in Akka much less reliable and less usable as such.

If we keep persisting events of an Actor into the storage but they are not actually persisted (due to some issue) without us knowing about this, we won’t be able to recover or re-initialize the Actor properly.

Also, if persistence is not guaranteed, we won’t be confident in data that we would stream out from event storage (for example with akka-projection) for further processing. Since event storage is our focal point for events data in Akka Persistence model, missing events in there would become a major issue.

2

Hi @amcontact,

The events are only applied to the model after they are persisted.
If, while persisting, there is an error, the event won’t be applied at all.

If we didn’t have that basic functionality Akka Persistence would be useless. We couldn’t even call it “persistence”.

I hope this clarifies.

Renato

3

Thank you for the response @octonato. Can there be a situation when Actor persisted the event without an error, however the event wasn’t eventually persisted eg due to some glitch on the storage end - with Cassandra?

4

No, this is not possible. When the actor persists the event, it means that is it stored on the DB. If something happen when doing that and the event is not persisted, the actor won’t apply that event.

5

Short correction, in Akka Persistence Typed, the event is applied before write, to verify that applying it does not throw an exception, then persisted. This is safe since the event application is meant to be side effect free. Any side effects are executed only after the event has been successfully persisted.

For the follow up question, if a Cassandra cluster nodes see catastrophic failure then data can be lost depending on consistency level configured, by default that is QUORUM which means that is quite unlikely but not impossible, but if you have tweaked it to for example ONE or LOCAL_ONE to improve performance, that highly increases that risk, if that one node accepted the write and then crash and burn before the write has been spread across the Cassandra cluster the write will have been lost without the actor noticing.