Weird Akka gRPC w/ Discovery behaviour

Hello - I’ve been trying to get config-based akka discovery setup for one of my gRPC services, but I’m struggling to make it work.

First off, when I do this, everything works fine:

val settings = 
  GrpcClientSettings
      .connectToServiceAt(<host>, <port>)
      .withSslContext(sslContext)
      .withTls(false)

After I changed it to:

val settings = 
  GrpcClientSettings
      .withConfig(<serviceName>)
      .withSslContext(sslContext)
      .withTls(false)

no requests are coming through to the service and I am seeing the following exception:

 io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
 #011at io.grpc.Status.asRuntimeException(Status.java:535)
 #011at akka.grpc.internal.UnaryCallAdapter.onClose(UnaryCallAdapter.scala:40)
 #011at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:557)
 #011at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:69)
 #011at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:738)
 #011at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:717)
 #011at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
 #011at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
 #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 #011at java.lang.Thread.run(Thread.java:748)```

I’ve been enabling netty logs

handlers=java.util.logging.ConsoleHandler
io.grpc.netty.level=FINE
io.grpc.ChannelLogger.level=FINE
java.util.logging.ConsoleHandler.level=FINE
java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter

but they do not show anything useful. This is my config:

akka {
  discovery {
    method = config

    config.services = {
      <serviceName> {
        endpoints = [
          {
            host = "<host>"
            port = <port>
          }
        ]
      }
    }
  }

  grpc.client."*" {
    service-discovery {
      mechanism = "config"
      protocol = "tcp"
      port-name = "https"
    }
  }

  grpc.client.<serviceName> {
    service-discovery.service-name = "<serviceName>"
    port = <port>
  }
}

I tried multiple variations all with the same effect and am running out of ideas of where to look. Here’s hoping you could shine a light on anything that looks suspicious in the above?

Actually I noticed a difference, which led to a separate issue and I think I know why, but I’m not sure if this is expected. I was mistaken in that the initial version did not have withTls(false), so I removed it. Now getting an Ssl exception:

Caused by: io.grpc.netty.shaded.io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching <serviceName> found.

I’m surprised to see the service name in the error message, instead of the host chosen by the discovery process. It may explain why this does not fail when using connectToServiceAt, which ends up (without config override) into the static | grpc-dns branch here:

val sd = serviceDiscoveryMechanism match {
      case "static" | "grpc-dns" =>
        val host = clientConfiguration.getString("host")
        require(host.nonEmpty, "host can't be empty when service-discovery-mechanism is set to static or grpc-dns")
        // Required by the Discovery infrastructure, set to host as we use static or grpc-dns discovery.
        serviceName = host
        staticServiceDiscovery(host, port)
      case other =>
        require(serviceName.nonEmpty, "Configuration must contain a service-name")
        Discovery(sys).loadServiceDiscovery(other)
    }

I haven’t got any further, I suppose I can add the service name in the certificate. Was just wondering if this was correct?