Need help understand how Play (with akka-http-core) can scale

I had posted a question (with great responses btw) about scaling a play-2.7 REST API with mongodb backend here if it helps to setup some context… Need help with load using akka.http or netty.http The original skeleton of the project was created from a play-scala-rest-sample with controller->handler->db-dispatcher->mongo-repository.

Could someone please explain how (from a very high level) play with akka (or netty) is supposed to scale to “millions of requests per second”. I do understand that at some point bigger hardware and perhaps some dispatcher tuning will help but what I am seeing just doesn’t make sense to me.

Scenario 1 - delay the controller response
In the play-scala-rest-api-example (https://github.com/bradrust/play-samples) I am “sleeping” a bit to simulate a service that might be slow. Maybe that isn’t the correct way to simulate something slow.

  def index: Action[AnyContent] = PostAction.async { implicit request =>
    logger.trace("index: ")
    for {
      _ <- Future.successful(logger.info("start"))
      m <- postResourceHandler.find
      _ <- pattern.after(2.seconds, system.scheduler)(Future.successful(Unit))
      _ <- Future.successful(logger.info("end"))
    } yield {
      Ok(Json.toJson(m))
    }
  }

But in that scenario, I can’t get an “ab” to succeed with concurrency of 300. Maybe this ab usage is wrong.
ab -c 300 -n 300 -s 120 localhost:9000/v1/posts
Which soon responds with
apr_socket_recv: Connection reset by peer (54)

Scenario 2 - my own dispatcher
I don’t have any code showing this but I have created my own dispatched and Actor configuration where from the controller I am doing an Ask. Inside of the actor I am doing the same “sleep” as above. Exactly the same results.

Here is my question I guess…
If my business-logic or handler is “slow”,… is there something that should help me out-of-the-box with Play 2.7 and akka-http (or netty)?

I understand there are A LOT of moving parts in large scale systems. I am just looking for some higher level explanation of “here is why play and akka scale” outside of the current Play marketing and documentation. Just seems like if my business-logic isn’t below 10ms then the only way to scale this is to pay a shit-ton of money for bigger hardware.

If it matters, I seem to get the same results across numerous configurations

  • prod-ish Google cloud, kubernetes cluster, openjdk:11 (tried lots of JDK’s), 2G, 4CPU
  • dev-ish MacOs, runProd or run, Zulu11JDK
  • dev-ish Windows 10, lots of RAM and Cores.

This post seemed to be related … How play backpressures akka http connection flow?

I do not write/read Scala, so I am going to base this on your description. So you are using “sleep” to simulate the code for doing something (e.g. round trip to the database), is the sleep on the HTTP thread or another thread from a different pool? I am just wondering if putting it to sleep is blocking/holding the thread from the HTTP thread pool.

Actually, I think some of what I am seeing could be from the way I am trying to test. So It did appear to me that the “sleep” was on one of the akka.http threads.

I have since tried to immediately send (ask) a message to an Actor pool to ensure it is usual a different thread pool for business logic and any “sleeps” I still get some very non-scalable (IMHO) connections per second.

Does anyone have a configuration they could share where they are doing low-latency, high-concurrency akka-http play apps?

I am guessing you are using your own dispatcher, i.e. scenario #2 from above? I have not created custom dispatchers myself, I use the default out of box, and set up a separate thread pool (execution context) for my blocking DB operations (the DB driver is blocking), by following the Play Threadpool documentation.