Akka cluster using Cassandra driver and running on EKS, with Keyspaces for Cassandra, generates weird warning

Akka Libraries
,
1

This WARN happens after an action is performed by the Cassandra driver after persisting data.

[2020-09-10 07:12:00,123] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-1] [] - [s0|/3.25.37.71:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0x6416a3c0, L:/10.62.160.46:38170 - R:3.25.37.71/3.25.37.71:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))
[2020-09-10 07:12:00,128] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-0] [] - [s0|/3.25.37.70:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0xfc77d13f, L:/10.62.160.46:55334 - R:3.25.37.70/3.25.37.70:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))
[2020-09-10 07:12:00,159] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-0] [] - [s0|/3.25.37.127:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0xaed46d98, L:/10.62.160.46:41326 - R:3.25.37.127/3.25.37.127:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))
[2020-09-10 07:12:00,203] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-1] [] - [s0|/3.25.37.66:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0xf8f9a736, L:/10.62.160.46:32856 - R:3.25.37.66/3.25.37.66:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))
[2020-09-10 07:12:00,387] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-1] [] - [s0|/3.25.37.126:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0x30782974, L:/10.62.160.46:53796 - R:3.25.37.126/3.25.37.126:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))
[2020-09-10 07:12:00,459] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-0] [] - [s0|/3.25.37.125:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0x740c3446, L:/10.62.160.46:44512 - R:3.25.37.125/3.25.37.125:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))
[2020-09-10 07:12:00,477] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-0] [] - [s0|/3.25.37.65:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0xd6a32bed, L:/10.62.160.46:58432 - R:3.25.37.65/3.25.37.65:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))
[2020-09-10 07:12:00,515] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-1] [] - [s0|/3.25.37.121:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0xbfff5e3c, L:/10.62.160.46:51852 - R:3.25.37.121/3.25.37.121:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))
[2020-09-10 07:12:03,776] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-1] [] - [s0|/3.25.37.71:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0xd5751f2b, L:/10.62.160.46:38222 - R:3.25.37.71/3.25.37.71:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))
[2020-09-10 07:12:04,002] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-1] [] - [s0|/3.25.37.126:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0x31832a99, L:/10.62.160.46:53842 - R:3.25.37.126/3.25.37.126:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))
[2020-09-10 07:12:04,112] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-0] [] - [s0|/3.25.37.125:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0x593bdfca, L:/10.62.160.46:44558 - R:3.25.37.125/3.25.37.125:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))
[2020-09-10 07:12:04,214] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-1] [] - [s0|/3.25.37.66:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0x74181e17, L:/10.62.160.46:32908 - R:3.25.37.66/3.25.37.66:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))
[2020-09-10 07:12:04,299] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-0] [] - [s0|/3.25.37.70:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0xd940b5f6, L:/10.62.160.46:55396 - R:3.25.37.70/3.25.37.70:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))
[2020-09-10 07:12:04,386] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-0] [] - [s0|/3.25.37.127:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0x0c030eca, L:/10.62.160.46:41386 - R:3.25.37.127/3.25.37.127:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))
[2020-09-10 07:12:04,608] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-1] [] - [s0|/3.25.37.121:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0xc87a74f8, L:/10.62.160.46:51902 - R:3.25.37.121/3.25.37.121:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))
[2020-09-10 07:12:04,657] [WARN] [] [com.datastax.oss.driver.internal.core.pool.ChannelPool] [s0-admin-0] [] - [s0|/3.25.37.65:9142]  Error while opening new channel (ConnectionInitException: [s0|id: 0x7fa12653, L:/10.62.160.46:58486 - R:3.25.37.65/3.25.37.65:9142] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.4.0, CLIENT_ID=e5c3c8f8-553e-4ffb-a753-f33678d6d15c}): failed to send request (javax.net.ssl.SSLException: SSLEngine closed already))

The data is still being persisted tough.

keyspaces_data
keyspaces_data2004×1127 169 KB

Also, used the suggested configuration by AWS : https://docs.aws.amazon.com/keyspaces/latest/devguide/programmatic.credentials.html#programmatic.credentials.SigV4_MCS

datastax-java-driver {

    basic.contact-points = [${?CASSANDRA_DNS_AND_PORT}]

    advanced.auth-provider{
        class = software.aws.mcs.auth.SigV4AuthProvider
        aws-region = ap-southeast-2
    }

    basic.load-balancing-policy {
        local-datacenter = "ap-southeast-2"
    }

    advanced.ssl-engine-factory {
        class = DefaultSslEngineFactory
        truststore-path = "/cassandra_truststore.jks"
        truststore-password = ${?CASSANDRA_TRUSTSTORE_PASSWORD}
    }
}

datastax-java-driver.profiles {
  akka-persistence-cassandra-profile {
    basic.request.consistency = LOCAL_QUORUM
  }
}
2

I couldn’t figure it out either why this error happens. In my case everything worked as expected so I’ve turned off this warning by setting:

    # Whether to log non-fatal errors when the driver tries to open a new connection.
    #
    # This error as recoverable, as the driver will try to reconnect according to the reconnection
    # policy. Therefore some users see them as unnecessary clutter in the logs. On the other hand,
    # those logs can be handy to debug a misbehaving node.
    #
    # Note that some type of errors are always logged, regardless of this option:
    # - protocol version mismatches (the node gets forced down)
    # - when the cluster name in system.local doesn't match the other nodes (the node gets forced
    #   down)
    # - authentication errors (will be retried)
   datastax-java-driver.advanced.connection.warn-on-init-error = false