You are creating too many HashedWheelTimer instances. HashedWheelTimer is a shared resource that must be reused across the JVM, so that only a few instances are created.
The error appears in the log after about a week of using the application in production.
The processor usage goes up about 1 percent each day. After a week it reaches 10% and won’t go down. I haven’t seen if it goes up further, since we are deploying a new version every week.
I can’t find what I’m doing wrong.
We are using play 2.8.7
Controller example:
public class MyController extends Controller {
final ActorSystem actorSystem;
final DatabaseExecutionContext dbEc;
@Inject
public MyController(ActorSystem actorSystem, DatabaseExecutionContext dbEc) {
this.actorSystem = actorSystem;
this.dbEc = dbEc;
}
public CompletionStage<Result> analysisList(){
return CompletableFuture.supplyAsync(()->
ok(Json.toJson(Analysis.find.all())),
HttpExecution.fromThread((Executor) dbEc));
}
public CompletionStage<Result> performLongRunningTask(Http.Request request){
final Map<String, String[]> params = request.body().asFormUrlEncoded();
if (params.containsKey("analysisId")) {
return CompletableFuture.supplyAsync(() -> {
Analysis item = Analysis.find.byId(params.get("analysisId")[0]);
if (item != null && item.isReady()) {
actorSystem.scheduler().scheduleOnce(Duration.ZERO, () -> longRunningTask(item), dbEc);
return ok("Task in process");
}
return ok("Item not ready");
}, HttpExecution.fromThread((Executor) dbEc));
}else
return CompletableFuture.completedFuture(badRequest("Missing parameter"));
}
private void longRunningTask(Analysis item){
//Do something that takes a long time to complete.
//High database usage.
}
public class DatabaseExecutionContext extends CustomExecutionContext {
@Inject
public DatabaseExecutionContext(ActorSystem actorSystem) {
super(actorSystem, "database.dispatcher");
}
If I remember correctly, HashedWheelTimer is an internal class to Netty. Review what netty usages are there on you application: it may be the Play’s HTTP backend (but the default HTTP backend in Play 2.8.x is Akka HTTP), it may be a database driver, or maybe you use a third party library which brings Netty as a transitive dependency.
It looks like you are creating too many instances, instead of reusing a single one or releasing the instances once used.
Why are you creating a new instance of the ActorSystem and the AhcWSClient for each instance of MyClass? You should be able to inject those into the class and share them across your entire application. Both the ActorSystem and AsyncHttpClient are expensive to create so you want to avoid creating many instances of them.
Thanks for your comments. MyModule is extending Abstract Module, so I couldn’t use @Inject. After some research I found that I could use @Provides on my method. The MyModule class now looks like this: