Need help with load using akka.http or netty.http

Akka Akka HTTP
1

Long and the short of it… reverting back to netty.http because I can’t get akka.http to satisfy a decent “load”.

I upgraded my project to play 2.7 and I’m having some immediate issues with load-testing. Unfortunately, I am not even getting close to advertised (just normal marketing and hype) load abilities against my application. I realize that there are a lot of factor involved when load-testing but my results are so poor I thought I’d ask for help and see if anyone had any suggestions.

Using a gatling simulation against my play 2.7 scala (exercising only two routes), application with a mongo (reactivemongo) backend. AFAICT, my asynchronous programming is correct.

As soon as I get up to around 100 requests/sec, I run into connection issues to the server. Issues like Premature close, Connection timed out, Connection reset by peer, etc… Prior to getting to failure, my typical response-time for a request in on the order of 5ms-10ms.

There doesn’t appear to be any significant load on my mongo backend. There is some load on the java process but nothing that seems outrageous.

I have tried some tuning of akka.http.server… config. I’ve tried some tuning of fork-join-executor. But never can seem to get to anything which seems to be a system/configuration that represents a scalable system. When running in a kubernetes cluster, I’ve done some horizontal and vertical scaling but it just seems like horrible numbers/cost-of-resources.

  1. doesn’t anyone have any template for “here is what worked best for me for a high connection rates using akka.http”?

  2. When I switch (back) to netty.http, I immediately get much better gatling/load numbers where I can at least identify that my contention is at the mongo part of the architecture. Does anyone have any gut opinions about akka.http should be more scalable than netty.http? Or what I should try to keep faith of the akka.http config.

  3. Based on a REST API -> play-json -> formats -> mongodb application (without significantly large model objects or complexity) … does anyone have any generic numbers about how many requests/sec the application should support on a 2-CPU, 4GB “machine”? (just taking one of the lower VM machine types in a cluster)

Thanks in advance,…
Brad

2

Have you tried forcing to use Akka Http 10.1.8 ?

3

Hi @bradrust,

indeed 100 requests/sec seem low. On that scale there should be no difference between akka-http and netty.

Have you tried profiling your application while the load test is running? Could you just do jstack <pid> while running the gatling test and post the output here?

Is your load test using persistent HTTP connections? Or is it opening new connections all the time?

The most common error that leads to bad performance is running into thread starvation because of running blocking code in your service handlers on the main dispatcher. Are you calling mongo with asynchronous APIs? If not, do you run the mongo calls on a dedicated dispatcher? A quick jstack can already confirm these kinds of issues.

A main reason the akka-http backend might show different behavior in that regard compared to netty is that the netty backend spawns its own thread pool. Altogether there might be more threads available for all the work to do and you might be able to achieve greater parallelism even in the presence of blocking code. With the akka-http backend there might only be the default dispatchers which has only limited threads, which is optimal if you don’t run blocking code. With blocking code, all the threads might be busy waiting for mongo and not being able to accept or handle any more HTTP connections.

So the first step is to confirm or rule out thread starvation issues.

Johannes

1 Like