Unhandled ChannelReadable messages received by deadLetters actor

Akka Libraries Akka HTTP
1

We have a REST server running on Akka for actors and incoming and outgoing HTTP requests and recently tried to upgrade from Akka 2.5.17 to 2.5.21 (and from Akka HTTP 10.1.3 to 10.1.7) and started receiving a lot of unhandled ChannelReadable messages in our logs from deadLetters actor. We were seeing these messages occasionally before but with that upgrade it went from maybe 10 a day to 1000 in an hour. Has anyone seen these messages and what is the cause of them? We don’t see the same volume in our development environment so it’s most likely on the incoming requests and it’s hard to pinpoint what is causing them. We could silence the log for these messages but need to ensure it’s not a real problem that we need to fix.

Example log entry (this is from our unhandled messages handler): “Unhandled message received by actor Actor[akka://AqutoServices/system/IO-TCP/selectors/$a/4904585#-595550038] from actor Actor[akka://AqutoServices/deadLetters]: ChannelReadable”

Any help/advice appreciated.

2

Thanks for the report, @padjoo. I found a likely culprit for those extra messages. They are probably due to a race condition (while shutting down TCP connections) that was introduced in https://github.com/akka/akka/commit/7899708ca684ce696fd3aa306cf56b396637b632 for Akka 2.5.19. They are harmless but can be noisy under circumstances that trigger the race condition.

I proposed a fix here: https://github.com/akka/akka/pull/26996

3

Thanks for the response and proposed fix @jrudolph. We’ll swallow those messages for now until the fix is ready. Just wanted to make sure it wasn’t a real issue.