Changing maximum response size (getting error RESOURCE EXHAUSTED: Compressed gRPC message exceeds maximum size 4194304)

Hi, great work on this library. I’m currently using the scala client and server both generated using this version 0.7.1. I’m occasionally getting errors like this returned from the service:

io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Compressed gRPC message exceeds maximum size 4194304: 4196022 bytes read

Here’s what it looks like in intelij’s debugger:

So assuming that’s in bytes, that’s a max response size of about 4 mb. It would be great if I could bump that up to about 32 mb. I’ve done some googling but I can’t figure out if this is a setting on the client or server side, and, more importantly, I’m not sure how I’d change it.

Appreciate any tips.

I am not sure if this helps but I just found this while reading the docs. Under the “Streaming service Client Configuration” it mentions changing the maximum size for messages. here is the docs link

Thanks @AkashicSeer. I’m glad you reminded me to post here.

I ended up figuring it out shortly after posting by digging into the underlying grpc-netty client setup.

Here’s how you’d set it to 32 mb using a channel builder override.

val settings = GrpcClientSettings
      .fromConfig(MyService.name)
      .withChannelBuilderOverrides(_.maxInboundMessageSize((32 * 1e6).toInt))
val client = MyServiceClient(settings)

I confirmed it respects the config by setting it to 1mb and making sure it crashed with an error message reflecting the change.

2 Likes

Thanks for posting this I am sure I will run into this problem soon.

Quick note: to adjust this same setting in Python you should do the following:

channel = insecure_channel("localhost:50051", options=[
    ('grpc.max_send_message_length', int(32e9)), # 32mb
    ('grpc.max_receive_message_length', int(32e9)), # 32mb
])
client = MyServiceStub(channel)

It’s important to cast the the value as int. Otherwise python’s scientific notation returns a float and grpc seems to ignore the setting.

Need similar help need for me as well. Below is the sample log of NettyClientHandler

14:51:19.332 [DEBUG] i.n.h.s.SslHandler - [id: 0x7af8407d, L:/ - R:] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

14:51:19.340 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] OUTBOUND SETTINGS: ack=false settings={ENABLE_PUSH=0, MAX_CONCURRENT_STREAMS=0, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192}

14:51:19.343 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041

14:51:19.346 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] INBOUND SETTINGS: ack=false settings={MAX_CONCURRENT_STREAMS=128, INITIAL_WINDOW_SIZE=65536, MAX_FRAME_SIZE=16777215}

14:51:19.348 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] OUTBOUND SETTINGS: ack=true

14:51:19.364 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] INBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=2147418112

Above is basic handshake and first RPC call from gRPC client


14:51:19.366 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] OUTBOUND HEADERS: streamId=3 headers=GrpcHttp2OutboundHeaders[********] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false

14:51:19.370 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] OUTBOUND DATA: streamId=3 padding=0 endStream=true length=30 bytes=00000000190a10089884f1d1832010b89286bf8320180412057669657731

14:51:19.673 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] INBOUND SETTINGS: ack=true

14:51:20.136 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] INBOUND HEADERS: streamId=3 headers=GrpcHttp2ResponseHeaders[*********] padding=0 endStream=false

14:51:20.148 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] INBOUND DATA: streamId=3 padding=0 endStream=false length=8192 bytes=0000020ce10ac0e6050a0963617465676f72793112610a16687770726f675f676873746d6b7231305f6d617331321234020d6b696e6773746f6e2d7465737409...

14:51:20.149 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] OUTBOUND PING: ack=false bytes=1234

As you can see in INBOUND DATA, I can see max of “length=8192” for each INBOUND_DATA even though response of a specific RPC is in size of 100MB . Is there any config available to toggle to higher frame size.

Is it possible to increase the size of each frame in gRPC response. Below is initial handshake from gRPC client to server and my client send first gRPC call

14:51:19.332 [DEBUG] i.n.h.s.SslHandler - [id: 0x7af8407d, L:/ - R:] HANDSHAKEN: protocol:TLSv1.2 cipher suite:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

14:51:19.340 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] OUTBOUND SETTINGS: ack=false settings={ENABLE_PUSH=0, MAX_CONCURRENT_STREAMS=0, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192}

14:51:19.343 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041

14:51:19.346 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] INBOUND SETTINGS: ack=false settings={MAX_CONCURRENT_STREAMS=128, INITIAL_WINDOW_SIZE=65536, MAX_FRAME_SIZE=16777215}

14:51:19.348 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] OUTBOUND SETTINGS: ack=true

14:51:19.364 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] INBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=2147418112


14:51:19.366 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] OUTBOUND HEADERS: streamId=3 headers=GrpcHttp2OutboundHeaders[********] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false

14:51:19.370 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] OUTBOUND DATA: streamId=3 padding=0 endStream=true length=30 bytes=00000000190a10089884f1d1832010b89286bf8320180412057669657731

14:51:19.673 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] INBOUND SETTINGS: ack=true

14:51:20.136 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] INBOUND HEADERS: streamId=3 headers=GrpcHttp2ResponseHeaders[*********] padding=0 endStream=false

14:51:20.148 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] INBOUND DATA: streamId=3 padding=0 endStream=false length=8192 bytes=0000020ce10ac0e6050a0963617465676f72793112610a16687770726f675f676873746d6b7231305f6d617331321234020d6b696e6773746f6e2d7465737409...

14:51:20.149 [DEBUG] i.g.n.NettyClientHandler - [id: 0x7af8407d, L:/ - R:] OUTBOUND PING: ack=false bytes=1234

On thing I want to change is “INBOUND DATA” length I receive from gRPC server to configurable. Even though gRPC response is 100M, my client is getting each frame as max of 8192 bytes