Meet with random "Unhandled error: [Stream with ID [1027] was closed by peer with code NO_ERROR(0x00)]"

Akka Akka gRPC
1

Hi,

I have below setup in our k8s production environment:

client → envoy engress → envoy ingress → grpc-server-app

and today one of the grpc-server-app pod throws 5xx errors for one seconds, and logs error Unhandled error: [Stream with ID [1027] was closed by peer with code NO_ERROR(0x00)]

And at the same time client side throws StatusRuntimeException msg=UNAVAILABLE: upstream connect error or disconnect/reset before headers. reset reason: overflow

Could anybody help share any insight how can I proceed my investigation? It seems related with connection management or buffer management.

2

I think it is something like your envoy proxy closing a HTTP/2 stream in a somewhat unexpected/abrupt way. Probably not critical. Not sure exactly what would cause envoy to do that, maybe the instance is crashing, shut down, or the connection/stream reached some timeout or limit inside envoy making it shut down with “overflow” as reason.

With Akka gRPC 2.3.0+ the peer-disconnect log is downgraded to a warning instead of an error on the server side, since it is not really critical.

3

Many thanks Johan for the insight! Yes I am now inspecting the envoy settings to if something is related.