Alpakka and S3 truncating downloaded files

Akka Akka Streams & Alpakka
1

Hello,

I have a simple piece of code, based on alpakka examples, which should download some file from S3 for further processing:

S3.download(bucket, file)
  .runWith(Sink.head)
  .flatMap {
    case Some((data, _)) =>
      data.map(_.utf8String).runWith(Sink.head).map(Some.apply)
    case None =>
      Future.successful(None)
    }

The problem is that the file content is getting truncated, the file size from the ObjectMetadata is correct it has ~2M it isn’t a huge file.

What I noticed is, when I use Sink.head the file content is from the beginning to middle if I change it to Sink.last it is from the middle to the end. Am I getting chunks from the file but why are they not seem to be streamed?

Can’t figure out what’s happening and how to fix this.

Thanks

2

Please disregard my question I’ve found the solution myself. In the end it was very clear…

Just need to replace:
data.map(_.utf8String).runWith(Sink.head).map(Some.apply)

with:
data.map(_.utf8String).runWith(Sink.seq).map(_.mkString).map(Some.apply)

accumullating all the chunks from the file.

Thanks

1 Like