Has anyone any experience with the new StreamRefs? I’m looking to use them in a cluster where a job offers a SourceRef to sharded workers and just wondering what happens if a worker node gets re-balanced, will the stream recover? Is there a backoff & retry?
Stream refs are absolutely expected to be sent over Akka remoting to other nodes within a cluster, and as such, complement and do not compete with plain Actor messaging. Actors would usually be used to establish the stream, by means of some initial message saying “I want to offer you many log elements (the stream ref)”, or alternatively in the opposite way “If you need to send me much data, here is the stream ref you can use to do so”.
What happens during failure after the initial offer of a stream ref…?
It’s only the StreamRef that can be serialized and sent to other nodes in the cluster. Once it’s materialized you have a local running stream that can’t be relocated. When your sharded actor is re-balanced (stopped) that stream has to end and you have to establish a new stream from another StreamRef from the new location.
Depending on the use case you might want to manage some offset to be able to continue from where you were. That is something you have to add yourself, in the actor message exchange, when retrieving the StreamRef from for a given offset.
Thanks for the answer!
Do you know how to build a resumable stream over StreamRef?
I try to use it in my project but sometimes the actor is rebalanced or passivated and the stream is failed.
I want to resume the stream which was failed, without rebuilding it, because I send this stream to a client, and it should be resumed for a client without breaking.
I am a newbie in Akka streams, perhaps this problem is easy to resolve.
Can you suggest me some solutions or any recommendations, please?
Thanks!