Alpakka - Bigquery.InsertAllAsync not working as expected

Hello,

Getting below exception while trying to write data into BigQuery using Alpakk’s Bigquery.InsertAllAsyc. function.

AkkaVersion = “2.6.19”
AlpakkaVersion = “3.0.4”
AkkaHttpVersion = “10.2.4”

{ "exceptionMessage" : "Error in stage [akka.http.impl.engine.client.OutgoingConnectionBlueprint$PrepareResponse$$anon$3]: HTTP message must not contain more than one Content-Type header " , "exceptionStacktrace" : "akka.http.scaladsl.model.IllegalResponseException: HTTP message must not contain more than one Content-Type header\n    at akka.http.impl.engine.client.OutgoingConnectionBlueprint$PrepareResponse$$anon$3.onPush(OutgoingConnectionBlueprint.scala:191)\n at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:542)\n   at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:423)\n   at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:650)\n    at akka.stream.impl.fusing.ActorGraphInterpreter$SimpleBoundaryEvent.execute(ActorGraphInterpreter.scala:61)\n  at akka.stream.impl.fusing.ActorGraphInterpreter$SimpleBoundaryEvent.execute$(ActorGraphInterpreter.scala:57)\n at akka.stream.impl.fusing.ActorGraphInterpreter$BatchingActorInputBoundary$OnNext.execute(ActorGraphInterpreter.scala:104)\n   at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:625)\n    at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:800)\n at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:818)\n  at akka.actor.Actor.aroundReceive(Actor.scala:537)\n    at akka.actor.Actor.aroundReceive$(Actor.scala:535)\n   at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:716)\n   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)\n   at akka.actor.ActorCell.invoke(ActorCell.scala:548)\n   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)\n    at akka.dispatch.Mailbox.run(Mailbox.scala:231)\n   at akka.dispatch.Mailbox.exec(Mailbox.scala:243)\n  at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)\n    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)\n    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)\n    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)\n" } 
{ "exceptionMessage" : "ErrorLog(8c4d1f48-25f1-4e60-825e-16f53a7ace2e,WMC_OP_CLR_INV,null,null,2023-03-31 10:10:07.939,2023-03-31 10:10:07.939,CILL_USER,enrich,CILL_USER,[Lcom.walmart.finance.cill.models.Entities$ErrorDetails;@6b1799b2,O,null,null,[Lcom.walmart.finance.cill.models.Entities$ErrorField;@38cdcc1e) " , "exceptionStacktrace" : "" } 
{ "exceptionMessage" : "Failed to process Offset :  Connection reset " , "exceptionStacktrace" : "" } 
{ "exceptionMessage" : "Failed to insert error log :  Connection reset " , "exceptionStacktrace" : "" } 
{ "exceptionMessage" : "Orchestrator: Stopped Stream. Throwable Exception - HTTP message must not contain more than one Content-Type header " , "exceptionStacktrace" : "akka.http.scaladsl.model.IllegalResponseException: HTTP message must not contain more than one Content-Type header\n   at akka.http.impl.engine.client.OutgoingConnectionBlueprint$PrepareResponse$$anon$3.onPush(OutgoingConnectionBlueprint.scala:191)\n at akka.stream.impl.fusing.GraphInterpreter.processPush(GraphInterpreter.scala:542)\n   at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:423)\n   at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:650)\n    at akka.stream.impl.fusing.ActorGraphInterpreter$SimpleBoundaryEvent.execute(ActorGraphInterpreter.scala:61)\n  at akka.stream.impl.fusing.ActorGraphInterpreter$SimpleBoundaryEvent.execute$(ActorGraphInterpreter.scala:57)\n at akka.stream.impl.fusing.ActorGraphInterpreter$BatchingActorInputBoundary$OnNext.execute(ActorGraphInterpreter.scala:104)\n   at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:625)\n    at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:800)\n at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:818)\n  at akka.actor.Actor.aroundReceive(Actor.scala:537)\n    at akka.actor.Actor.aroundReceive$(Actor.scala:535)\n   at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:716)\n   at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)\n   at akka.actor.ActorCell.invoke(ActorCell.scala:548)\n   at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)\n    at akka.dispatch.Mailbox.run(Mailbox.scala:231)\n   at akka.dispatch.Mailbox.exec(Mailbox.scala:243)\n  at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)\n    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)\n    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)\n    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)\n" } 
{ "exceptionMessage" : "Data write and Offset update Failed :   Offset Exception occurred : Exception occurred during offset insertion " , "exceptionStacktrace" : "" } 
[03-31-2023 15:40:08.887] [info] [com.walmart.finance.cill.utils.SinkOnCompleteHandler] [application-akka.actor.default-dispatcher-8] - Audit status post stop after sftp callback  FAILURE 
[03-31-2023 15:40:08.890] [info] [com.walmart.finance.cill.utils.SinkOnCompleteHandler] [application-akka.actor.default-dispatcher-8] - Downstream onCompleteAction successful
{ "exceptionMessage" : "ErrorLog(d9d45d9f-3c17-4a42-a075-acd6ee7b312d,WMC_OP_CLR_INV,null,null,2023-03-31 10:10:08.89,2023-03-31 10:10:08.89,CILL_USER,enrich,CILL_USER,[Lcom.walmart.finance.cill.models.Entities$ErrorDetails;@3a53a69,O,null,null,[Lcom.walmart.finance.cill.models.Entities$ErrorField;@58219d35) " , "exceptionStacktrace" : "" } 
{ "exceptionMessage" : "Failed to insert error log :  Connection reset " , "exceptionStacktrace" : "" } 
[03-31-2023 15:40:09.145] [info] [com.walmart.finance.cill.streams.DownstreamFlow] [application-akka.actor.default-dispatcher-24] - Scheduling Graph restart after : 1800 seconds, Graph restart attempt: #0

Can anyone please suggest a way to solve this issue.

Lots of

Connection reset

Indicates like some network or server connectivity issue to me.