2.5.23/10.1.9: Trying HTTP2, always getting a HTTP 503 error

I’m trying to enable HTTP2; took a synchronous
Http()(system) .bindAndHandle(RouteResult.route2HandlerFlow(route) to
Http2()(system) .bindAndHandleAsync(Route.asyncHandler(route) and now all my routes are turning a HTTP 503 error. I even tried wrapping my rout inside but that didn’t help other.
withoutRequestTimeout{
apiRoute
}

How can I tell what’s going on? Thanks much

BTW, I’m using JDK 8 and so use the java agent to enable H2.

Hi @surendarchandra,

are there any errors in the logs?

Can you try using the supported way of enabling Http2 as documented here: https://doc.akka.io/docs/akka-http/current/server-side/http2.html#server-side-http-2-preview-

I.e. set

akka.http.server.preview.enable-http2 = on

and use the normal

Http().bindAndHandleAsync()

not the Http2 variant?

Johannes

Thanks @jrudolph, there were no errors in the logs. If I strictly follow the instructions, Intellij complains about unused Http2 in

import akka.http.scaladsl.Http2
```. Once that import is removed, it strictly falls back to Http/1.1. It only negotiates Http2 is I chose Htttp2()

Even tried just calling Http2() (and so Scala can think it is in use but call Http() and it still only negotiates Http1.1. I specify the enable-http2 via application.conf and

  • private val customConf =
  • ConfigFactory
  • .parseString(
  •  """
    
  •    akka.http.server.preview.enable-http2 = on
    
  •    my-blocking-dispatcher {
    
  •      type = Dispatcher
    
  •      executor = "thread-pool-executor"
    
  •      thread-pool-executor {
    
  •        fixed-pool-size = 32
    
  •      }
    
  •      throughput = 1
    
  •    }
    
  •  """)
    
  • implicit val system = ActorSystem(“my-system”, ConfigFactory.load(customConf))

Just staring at the logs, I see this, request was using HTTP/2.0 but the response was using HTTP/1.1. Not sure how thet changed but could it be the issue?

Request minus entity: HttpRequest(HttpMethod(GET),https://10.0.120.148/api/v1/cluster/me/version,Vector(User-Agent: curl/7.58.0, Accept: /, Remote-Address: 10.0.64.32:48700),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/2.0))

withRequestTimeout was used in route however no request-timeout is set!

Response minus entity: HttpResponse(503 Service Unavailable,List(Cache-Control: private,),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))

Ah, right the HttpResponse model says HTTP/1.1 but it will still go out over HTTP/2 if it’s on an HTTP/2 connection.

Try enable debug logging using akka.loglevel = debug and post relevant logs from akka-http.

Note that these were with Http() (and not Http2() - had to set enable-http2 = true in the reference.conf.

[DEBUG] [07/25/2019 17:29:35.549] [my-system-akka.actor.default-dispatcher-5] [akka://my-system/system/IO-TCP/selectors/$a/1] New connection accepted

[DEBUG] [07/25/2019 17:29:35.634] [my-system-akka.actor.default-dispatcher-18] [Http2ServerDemux(akka://my-system)] Changing state from Idle to WaitingForNetworkToSendControlFrames

[DEBUG] [07/25/2019 17:29:35.636] [my-system-akka.actor.default-dispatcher-17] [Http2ServerDemux(akka://my-system)] Changing state from WaitingForNetworkToSendControlFramesto Idle

[DEBUG] [07/25/2019 17:29:35.637] [my-system-akka.actor.default-dispatcher-17] [Http2ServerDemux(akka://my-system)] Changing state from Idle to WaitingForData

[DEBUG] [07/25/2019 17:29:35.638] [my-system-akka.actor.default-dispatcher-16] [Http2ServerDemux(akka://my-system)] Got 3 settings!

[DEBUG] [07/25/2019 17:29:35.638] [my-system-akka.actor.default-dispatcher-16] [Http2ServerDemux(akka://my-system)] Setting max concurrent streams to 100 (not enforced)

[DEBUG] [07/25/2019 17:29:35.638] [my-system-akka.actor.default-dispatcher-16] [Http2ServerDemux(akka://my-system)] Setting initial window to 1073741824

[DEBUG] [07/25/2019 17:29:35.638] [my-system-akka.actor.default-dispatcher-16] [Http2ServerDemux(akka://my-system)] Ignoring setting SETTINGS_ENABLE_PUSH -> 0 (in Demux)

[DEBUG] [07/25/2019 17:29:35.638] [my-system-akka.actor.default-dispatcher-16] [Http2ServerDemux(akka://my-system)] Changing state from WaitingForData to Idle

2019-07-25T17:29:35.639+0000 DEBUG <.my-system-akka.actor.default-dispatcher-16> [spray.SprayServer] Request minus entity: HttpRequest(HttpMethod(GET),https://10.0.120.148\

/api/v1/cluster/me/version,Vector(User-Agent: curl/7.58.0, Accept: /, Remote-Address: 10.0.64.32:49444),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/2.0))

[DEBUG] [07/25/2019 17:29:35.638] [my-system-akka.actor.default-dispatcher-16] [Http2ServerDemux(akka://my-system)] Changing state from Idle to WaitingForData

[DEBUG] [07/25/2019 17:29:35.638] [my-system-akka.actor.default-dispatcher-16] [Http2ServerDemux(akka://my-system)] Updating outgoing connection window by 1073676289 to 1073741824

[WARN] [07/25/2019 17:29:35.639] [my-system-akka.actor.default-dispatcher-16] [akka.actor.ActorSystemImpl(my-system)] withRequestTimeout was used in route however no request-timeout is set!

2019-07-25T17:29:35.653+0000 DEBUG <.my-system-akka.actor.default-dispatcher-16> [spray.SprayServer] Response minus entity: HttpResponse(503 Service Unavailable,List(),HttpEntity.Strict(none/none,ByteString()),HttpProtocol(HTTP/1.1))

2019-07-25T17:29:35.654+0000 DEBUG <.my-system-akka.actor.default-dispatcher-16> [spray.SprayServer] Failed response entity: HttpEntity.Strict(text/plain; charset=UTF-8,/cluster/{id}/version is not available)

2019-07-25T17:29:35.654+0000 INFO <.my-system-akka.actor.default-dispatcher-16> [spray.SprayServer] HttpMethod(GET) https://10.0.120.148/api/v1/cluster/me/version -> 503 Service Unavailable

It says /cluster/{id}/version is not available) which doesn’t seem to be an akka-http error. It’s not quite clear that this would be related to HTTP/2 or akka-http.

Johannes

We use swagger to build that part. But the same code works for HTTP1.1, it only fails for HTTP2 and we don’t do anything regarding 1.1 or 2.0. I wasn’t even sure where we got that HTTP response protocol set to 1.1

Is it related to https://github.com/akka/akka-http/issues/952 ? The signature seems to match.

I do have
request-timeout = 300 s
in application.conf.

Not sure where I’m calling withRequestTimeout without a value. If it is not specified, do we use the default?

Found the problem, my route calculation logic was throwing a exception because it couldn’t find the Host header; switching it to optionalHeaderValueByName fixed it!!

Two questions:

  1. What is the best way to get Host or Authority header? In HTTP/1.1, I could get headerValueByName of headers.Host.name, is there a nice way to look for the Authority header other than two optional lookups and then finding the matching one?
  2. With bindAndHandleAsync(Route.asyncHandler(route), do I have to explicitly use a Future in my route calculation or it is already in a Future via the asyncHandler (sorry, still learning Scala). Thanks much @jrudolph