Exceeded configured max-open-requests value of [32]

Play Framework
,
1

I use Alpakka S3 to stream data to and from storage. In application.conf I have akka.http.host-connection-pool.max-open-requests = 128, but during stress testing, I get an “Exceeded configured max-open-requests value of [32]” in log. Why 32 if I set 128? How to fix it?

2

How did you execute the stress test? Do you override the akka-http version from Play?

3

By sending a large number of simultaneous HTTP requests to the API of my web service.

Yes, I added "com.typesafe.akka" %% "akka-http" % 10.1.12 to build.sbt But I use Play 2.7.9 so akka-http must be the same version.

4

Can you please post the code where you call the alpakka s3 api?

5

I tried to reproduce your problem, but I am not able to.
For me it seems

akka {
  http {
    host-connection-pool {
      max-open-requests = 7
    }
  }
}

was picked up it seems.

6

Also here another post: Questions about akka-http, alpakka s3 in Play

It seems there the max-open-requests config with the value 1024 was picked up correctly as well…

7

I checked under the debugger, the values in the loaded config are correct. But

(HostConnectionPoolSetup(some.domain.ru,7480,ConnectionPoolSetup(ConnectionPoolSettings(4,0,5,32,1,Duration.Inf,100 milliseconds,2 minutes,30 seconds,ClientConnectionSettings(Some(User-Agent: akka-http/10.1.12), …

It looks like something overrides configuration values. And I can not find where it occurs :frowning_face:

8

Based on your previous posts, you are using Play Java?
So when you call alpakkas S3 api you probably have to use a Materializer somehwere. How to you create that materializer? Do you inject it via Guice?

What akka-http does it creates the connection pool from the config of the actorsystem from the materializer. So if we know how you create the materializer we maybe can find out what the problem is.

Also what you can do as well is you can add a breakpoint at akka-http/ConnectionPoolSettingsImpl.scala at v10.1.12 · akka/akka-http · GitHub
and then look which methods called that line, and walk up until you find out where the config comes from.

BUT BE AWARE:
If you run Play in dev mode, as soon as you hit run, Play starts a dev http server (akka-http or netty). For this dev http server (no matter if akka-http or netty) an own dev actor system gets created (again: only in dev mode!). When using akka-http (the default), as soon as it starts (immediately after run) the breakpoint will be reached, and the value will be 32. That is OK. Just skip that for now. (That is happening because akka-http actually uses the mentioned dev actor system for binding ports which triggers the creation of the pools it seems. Also the value is 32 because at the time the dev http server starts no app has been started yet which usually reads application.conf and passes that read configs it to the server, but no application, no application.conf got read yet, so the dev server doesn’t know about its configs).
You need to run an actual request that triggers an S3 api call, only then the application actor system will create its connection pools and then the breakpoint will be triggered again. That’s what you want to investige.

1 Like
9

Yes.

No, I only return a chunked response with the source gets from S3. Play materialize if by itself.

I have prepared a minimal reproducible example.

And I tried to debug the initialization of ConnectionPoolSetting. The breakpoint did hit when the server started, as you said, but it didn’t fire anymore. After that, I ran the stress test in production and got no errors! It looks like the problem is with the dev server.

10

I forked your project and started debugging, because I was very interested what is going on.
It turns out for the upload, the breakpoint does get triggered, but for the download it does not…
So for the upload the correct config is used, but for the download it is not…
(Note we are just talking about dev mode here, in prod everything works, also for download)
I documented my research in the README of my fork, for me now everything is clear: GitHub - mkurz/play-s3-example: Example of problem with max-open-requests in dev mode

What you can do, so the config gets used in dev mode when using the akka-http backend, is to set

PlayKeys.devSettings ++= Seq(
  "akka.http.host-connection-pool.max-open-requests" -> "256"
)

in build.sbt.

1 Like
11

It’s an awesome answer! Thank you very much!

1 Like