I had posted a question (with great responses btw) about scaling a play-2.7 REST API with mongodb backend here if it helps to setup some context… Need help with load using akka.http or netty.http The original skeleton of the project was created from a play-scala-rest-sample with controller->handler->db-dispatcher->mongo-repository.
Could someone please explain how (from a very high level) play with akka (or netty) is supposed to scale to “millions of requests per second”. I do understand that at some point bigger hardware and perhaps some dispatcher tuning will help but what I am seeing just doesn’t make sense to me.
Scenario 1 - delay the controller response
In the play-scala-rest-api-example (https://github.com/bradrust/play-samples) I am “sleeping” a bit to simulate a service that might be slow. Maybe that isn’t the correct way to simulate something slow.
def index: Action[AnyContent] = PostAction.async { implicit request =>
logger.trace("index: ")
for {
_ <- Future.successful(logger.info("start"))
m <- postResourceHandler.find
_ <- pattern.after(2.seconds, system.scheduler)(Future.successful(Unit))
_ <- Future.successful(logger.info("end"))
} yield {
Ok(Json.toJson(m))
}
}
But in that scenario, I can’t get an “ab” to succeed with concurrency of 300. Maybe this ab usage is wrong.
ab -c 300 -n 300 -s 120 localhost:9000/v1/posts
Which soon responds with
apr_socket_recv: Connection reset by peer (54)
Scenario 2 - my own dispatcher
I don’t have any code showing this but I have created my own dispatched and Actor configuration where from the controller I am doing an Ask. Inside of the actor I am doing the same “sleep” as above. Exactly the same results.
Here is my question I guess…
If my business-logic or handler is “slow”,… is there something that should help me out-of-the-box with Play 2.7 and akka-http (or netty)?
I understand there are A LOT of moving parts in large scale systems. I am just looking for some higher level explanation of “here is why play and akka scale” outside of the current Play marketing and documentation. Just seems like if my business-logic isn’t below 10ms then the only way to scale this is to pay a shit-ton of money for bigger hardware.
If it matters, I seem to get the same results across numerous configurations
- prod-ish Google cloud, kubernetes cluster, openjdk:11 (tried lots of JDK’s), 2G, 4CPU
- dev-ish MacOs, runProd or run, Zulu11JDK
- dev-ish Windows 10, lots of RAM and Cores.
This post seemed to be related … How play backpressures akka http connection flow?