ERROR [akka.actor.ActorSystemImpl] Uncaught error from thread [mpe-akka.remote.default-remote-dispatcher-6]: null, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[mpe] java.lang.StackOverflowError: null

mdasif114 · February 14, 2019, 5:59am

Hi All,

I am working on a fix which needs to accept a SOAP XML Request which is less than 1,00,000 threshold value. So that we can validate large xml documents before processing.

Now, I am confused about the error reporting by akka when the threshold value is less than 1,00,000. I noticed the if the elements under one parent is more than 2000 I am getting this error. if the elements are divided with multiple parent elements we are not getting the error. Please help me to figure out the below the error from akka.

I have also to tried with disabling "akka.jvm-exit-on-fatal-error" in the configuration.

akka {
  
  jvm-exit-on-fatal-error = off
  
  actor {
    provider = "akka.remote.RemoteActorRefProvider"
  }

  remote {
    enabled-transports = ["akka.remote.netty.ssl"] 
    .
    .
    .
  }
}

Example of Soap Request Body:

<soapenv:Body>
	<cai3:Create>
		<cai3:Attributes>
			<CreateAttributes attributeId="1">
				<attributeId>12345678</attributeId>
				<!--Zero or more repetitions:-->
				<node1 attributeId="12345678">
					<node1>12345678</node1>
					<node1/>
				</node1> 
				.......
				.......
				.......
				.......
				.......
				.......
				.......

				on so on upto 3000 same elements
				
				</CreateAttributes>
		</cai3:Attributes>
	</cai3:Create>
</soapenv:Body>

Exception Log

Module 2019-02-13 17:07:20,669+0800 [t-dispatcher-16] ERROR [akka.actor.ActorSystemImpl] Uncaught error from thread [mpe-akka.remote.default-remote-dispatcher-6]: null, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[mpe]
java.lang.StackOverflowError: null
        at java.lang.Exception.<init>(Exception.java:76) [na:1.8.0_172]
        at java.lang.ReflectiveOperationException.<init>(ReflectiveOperationException.java:89) [na:1.8.0_172]
        at java.lang.reflect.InvocationTargetException.<init>(InvocationTargetException.java:72) [na:1.8.0_172]
        at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source) [na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [na:1.8.0_172]
        at java.lang.reflect.Method.invoke(Method.java:498) [na:1.8.0_172]
        at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1128) [na:1.8.0_172]
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) [na:1.8.0_172]
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) [na:1.8.0_172]
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) [na:1.8.0_172]
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) [na:1.8.0_172]
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) [na:1.8.0_172]
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) [na:1.8.0_172]
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) [na:1.8.0_172]
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) [na:1.8.0_172]
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) [na:1.8.0_172]
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) [na:1.8.0_172]
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) [na:1.8.0_172]
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) [na:1.8.0_172]
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) [na:1.8.0_172]
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) [na:1.8.0_172]
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) [na:1.8.0_172]
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) [na:1.8.0_172]

TimMoore · February 14, 2019, 6:13am

This isn’t really an error from Akka, it’s an error from code that is called from an Akka dispatcher thread. It looks like you are serializing a deeply nested object graph and that the call stack is too deep for your JVM settings. Either there is a bug causing infinite recursion, or more likely, it is finite but just larger than the default stack size setting for the JVM, which you can increase with the -Xss parameter to java. For example, -Xss 2048k. You’ll need to test to find the exact size you need.

The stack trace is truncated, so it’s hard to know exactly what’s happening, but if you are serializing a large object to send as an Akka message, please note two recommendations:

Avoid very large objects in Akka remote messaging, since these are expensive to serialize and deserialize and can cause throughput issues
Java serialization should be avoided in general, in favor of faster and more secure options such as protobuf, avro, or even JSON. See https://manuel.bernhardt.io/2018/07/20/akka-anti-patterns-java-serialization/ for an in-depth explanation

mdasif114 · February 15, 2019, 6:01am

Hi Tim,

Thank you so much for your prompt response. I have tried with increasing JVM stack size, but it didn’t work for me.

It seems like the problem is with Java serialization. I will look into the link which is provided by you.

Best Regards,
Mohammed Asif

TimMoore · February 18, 2019, 4:15am

One possibility is that if the in-memory representation of the list of child elements is a linked list, the Java serializer would have to allocate a stack size proportional to the length of the list. You might be able to work around this more easily by using an array-backed list rather than a linked list. I’m not sure whether you’re using Java or Scala for this, or how the XML is being deserialized… the details would depend on these things. Can you post your code?

mdasif114 · February 21, 2019, 7:01am

Hi,

We are using both Java and Scala in our application. The issue is occurring while serializing the large objects in akka module (Scala code). So, we have to decided to validate the bad requests before it reaches to scala code.

Validate the size of request (In some cases, if the xml is not in well-formatted way, it will fail). So, we need below level check too.
Allow child elements less than configured amount of entries for each element (we configured the entries below 1000).

If the validation is failed at these two checks, we reject the request.

Thanks,
Asif

Topic		Replies	Views
Getting 'LocalActorRefProvider(akka://APIX) : guardian failed, shutting down system' Error Actors akka	0	588	October 15, 2022
[error] akka.actor.OneForOneStrategy [sourceThread=application-akka.actor.default-dispatcher Akka Libraries	0	924	November 11, 2020
Improving Resiliency of Akka IO DNS Akka Libraries	1	360	April 20, 2023
Guardian failure gives exit 0 on jvm termination Akka Libraries	2	341	April 1, 2022
Throwing ActorKilledException to stop singleton cluster actor Akka Libraries	8	791	June 28, 2018

ERROR [akka.actor.ActorSystemImpl] Uncaught error from thread [mpe-akka.remote.default-remote-dispatcher-6]: null, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[mpe] java.lang.StackOverflowError: null

Example of Soap Request Body:

Exception Log

Related topics