New to Alpakka, question about downloading file(s)

Akka Akka Streams & Alpakka
1

Hello,

I have been trying to figure a way to download a file from s3. I want to:

  1. list the files from s3://bucket/key
  2. filter for csv files.
  3. write them to a local directory

I have been working with this code but can’t get it to work. I am unfamiliar with how graph works. Can someone point me in the right direction.

  val test = S3.listBucket(bucket, Option(key))
    .filter(a => a.key.endsWith("csv"))
    .map(r => S3.download(r.bucketName, r.key))
    .to(FileIO.toPath(downloadPath))
    .run()

I am trying to make it a Future[Done] so I can call on complete to it and close out the system.

This code works but I dont know how to close out the system after running the graph to completion

  val test = s3List
    .filter(a => a.key.endsWith("csv"))
    .map(r => S3.download(r.bucketName, r.key))
    .flatMapConcat(identity)
    .collect{case Some(x) => x}
    .flatMapConcat{case (s, o) => s}
    .to(FileIO.toPath(downloadPath))

  test.run()
2

Hi @moyphilip, welcome to the community.

What do you mean by “close out the system” ?

3

i’ve been doing this to “close” @VictorBLS

  implicit val system = ActorSystem("test")
  implicit val materializer = ActorMaterializer()
  implicit val ec = system.dispatcher

      val test = s3List
        .filter(a => a.key.endsWith("csv"))
          .runForeach(println)
      
      test onComplete {
        case Success(_) => system.terminate()
        case Failure(t) => println("An error has occurred: " + t.getMessage)
      }
4

@moyphilip I mean what do you really want by “closing” the actor system ?

5

I want to exit my program after i finish writing the files, which i don’t know how to do.

6

If your system if not shutting down so it means that your stream is not being completed with a success try to terminate your system regardless of the result