We’re using HTTP2. When there are multiple GETs a few POSTs,we sometimes run into a scenario where a POST gets stuck waiting for entity for over 30 seconds. Since I’ve configured the limit to be 30 seconds, it triggers a HTTP 408 error for that POST request which unfreezees the rest of the pipeline. I don’t think 30 sconds is too low, it feels like I’m running out of threads to process this entity and some how the timeout frees up some thread for the other POST messages (which doesn’t make sense). Any ideas? Thanks much
I’m using default-dispatcher. Is that the problem (dispatcher already had 64 max threads and extract entity needs one more to do actual work?)
If you don’t block then the number of threads is not necessarily related. Can you check that you have no blocking code? E.g. using jstack <pid> on the shell.
If all these requests are running on the same connections it might also be other limits coming into play like the parallelism setting and potential restrictions on the number of concurrent requests per connection. Do you know why the POST request is stuck? Is it just a slow upload?
If requests on multiple connections are creating that situation then a thread starvation / blocking issue is more likely.
It is fleeting and so haven’t been able to capture the exact moment using jstack.
So, we have two sets of thread pools: default dispatchers and a separate pool for processing blocking operations like extractStrictEntity. What seems to be happening is Chrome opens many requests mostly GET and a few POST. Nothing is sent on the POST channel (all the servers are in same data center with 10Gbps links). And so it waits. Inexplicably, all other GETs also wait. Once thee entity processing times out, all the GETs proceed. Its not clear to me why the post processing, even if the browser is not sending anything, is affecting the GETs . Cannot figures out whether there is a browser issue or akka Side issue. There is no “lock” when processing posts right? I saw some mention that it will stop all stream processing and so was curious. Thanks much
You could try to enable DEBUG logging and maybe also akka.http.server.http2.log-frames = true but that will produce a lot of logging, so only enable it during a reproduction (if it’s reproducible).
You can also do the same on the browser side and enable logging on the about:net-internals page of chrome. This might help to see what’s going on on the line.
If you are able to create a reproducer, we can check whether it’s a bug or something else.