Akka Persistence Db Query error during AWS Instance type upgrade

Hi,

We upgraded our AWS RDS instance type that we use for our Akka Persistence Db. We use Slick jdbc driver on top. The steps are: 1. Spinup new db type and set as Read replica of the current Persistence Db. 2. Failover to the Read replica. The problem we encountered is that the persistence queries were failing because the MySQL server was running with --read-only flag. Why would this happen when the read replica should have been promoted to a write replica?

Our theory is that due to DNS caching Akka is still pointing to the old replica which has been downgraded to a read replica due to the failover. Is this what happens? How can we prevent this such that during failover persistence queries go to the new write replica?

Additional info on this. It seems to be related to the DNS caching of the Akka connection to the Persistence db. We tried setting JAVA_OPTS -Dnetworkaddress.cache.ttl=5 and we’re still encountering the same problem. Is there an Akka config to set the DNS cache ttl for the persistence db connection?

I’d expect that resolution is not done by the Akka DNS infra but by the JDBC driver itself using the JVM DNS APIs.

I think it is not possible to set networkaddress.cache.ttl via a system property since it is a special “security” property. Search for doing that to find the right way to set it to try out if that helps.