Considering we persist events for our actors/entities in event journal (in Cassandra) and periodically open a stream from persistent query (read journal also in Cassandra) which is querying events over a past time window, how can we ensure we won’t lose messages when consuming events from persistent query? These events are critical not to lose in downstream processing from persistent query and it would be important to ensure delivery guarantees (at least once).
Hi @amcontact,
it is hard to tell what guarantees you code offers if you are using custom query or how you control this “past time window”.
Instead of reinventing the wheel, I suggest you looked at the soon-to-be-released akka-projection. This new library in the Akka ecosystem provides implementations for some common use-cases such as projection the contents of an Akka Persistence Journal using at-least-once semantics.
Cheers,
Thank you @ignasi35. By delivery guarantees I mean the following: It’s discouraged for an Actor to publish events directly to some Custom_Store (eg in Kafka), in addition to events persisted in the Event_Journal (eg in Cassandra). And per my understanding, the primary way to transport data from Event_Journal to Custom_Store is by using Akka Persistence Queries. It’s not clear to me though if Persistence Query can provide “at least once” delivery guarantee when it makes data from Event_Journal available for recording into Custom_Store, when we can’t afford to miss messages in Custom_Store. This is quite critical as can be a weak link in reliable message delivery when we want to stream all events from persistent Akka Actors to other services in our solution.
Hi @amcontact,
what I meant is that without knowing which of the predefined queries and what type of offset upgrade strategy you will use it’s hard to discuss what your guarantees may be.
But I think I see your current stage in the development phase and I would recommend you didnt’ reinvent the wheel and had a look at akka-prokection which solved exactly the type of use cases you are trying to implement: given events on an akka persistence journal, emit events to a Kafka topic using at-least-once semantics.
Cheers,
1 Like
As using at-least-once semantics, Akka-projection seems to be what we’re looking for, do you know when it’s expected to be released and how production-ready it would be? It’s in Scala, would still be usable from Java?
PS The predefined query could be EventsByTag. We would like to stream all events at some point but such option seems to be not available.
The implementation is mostly in scala but there is a Java and a Scala APIs. We’re aiming at a 1.0.0 Release Candidate release by the end of this week.
You can consume all events using the eventsByTag query ~is~ if all events are tagged using the same tag. ;-)
(edited to fix a typo)
1 Like
Thank you @ignasi35 !