Read-side failure reporting

Lagom Lagom Persistence API
1

My team heavily utilizes Lagom’s ReadSideProcessors to create non-primary identifier lookups for our entities. It is an awesome feature that improves our write-side performance, as well as our assurance of delivery.

Regarding the assurance of delivery though, we put some relatively less-reliable processing in read side processors, specifically so that they get retried when they fail. Currently, it seems that when we bubble up the exception (or more likely failed CompletableFuture), it is logged by akka.stream.scaladsl.RestartWithBackoffSource as and ERROR with message “Restarting graph due to failure”. Considering this is a place that we expect to have some level of failures, we’d like to not report them as severely as other errors in the system (i.e. warn, or at least let the RSP report in some way the level of failure).

This logging isn’t in a Read side processor related class, or a Lagom class at all, so we are hesitant to ignore all akka.stream.scaladsl.RestartWithBackoffSource errors. We could write some log-mapper that at least translates any RestartWithBackoffSource ERRORs to a warning, but that still concerns me for the same reason, just a little less.

I’m not sure there is a way around it if the RSP implementation wants to utilize RestartWithBackoffSource though, and I understand why that is a perfect tool for it.

2

Hi @zmarois,

this is a great request and I think there’s a possible simple solution adding log on the framework code that runs the Flow of the read side processor.

Would you like to contribute an improvement?

3

I’m not sure that is solving the problem. It sounds like the logging of the RestartSource should be configurable (probably with a streams attribute) and then that can be used by Lagom’s RSP.

Please create an issue i. akka/akka as the first step.

4

Thank @patriknw for helping me portray my dilemma here. And thanks for confirming that I’m not missing some approach I could just take in Lagom.

Yes, that is exactly my problem. I’d love to contribute this back. I’ll open something in akka/akka regarding this, and if I can contribute back there (and then back to Lagom to consume it), I absolutely will.

5

Oh wait, I don’t think I understood your suggestion the first time @ignasi35. The log link you provided makes it sound like I can modify the akka stream log levels in Lagom. That’s sounds like it fixes the problem.

I’ll take a crack at it in the ReadSideActor you pointed to. Thanks for the akka knowledge.

6

The log operator adds additional logging, not changing logging (levels) in other operators or internals.

7

Thanks for saving me the time playing around with the log operator @patriknw.

Filed https://github.com/akka/akka/issues/25019. I’ll gladly make that contribution to Akka

8

Pushed this reduction from error to warning in akka. Should be available next time it is released