Akka streams join by key

Hi,

Using Akka-streams, what are my options for performing an inner-join (join-by-key), as described by Tyler Akidau in his Streaming 101 blog article?

I fee this should be rather straightforward, but cycling through the standard operators, custom GraphStages and Actors I can’t figure out which is the best approach.

Some advice would be much appreciated!

Hi Mark. Join by key unfortunately has no high level support in the Akka Streams Operator DSL, but it is easy to merge and zip together streams into a single combined stream, which can then have be aggregated in some form to implement join by key. Sliding window operations would make this effort easier, but they don’t have an easy to use API either (though it’s something we’re considering). Before diving into Custom Graph Stages I would recommend looking at the statefulMapConcat operator. You could maintain state within this stage that can be used to aggregate a sliding window of a merged stream over time and to do the appropriate key lookups. Because it has mapConcat semantics (i.e. flatMap), you can optionally emit elements downstream for each element received. To recreate a sliding window you could also merge in another stream that sends “ticks” (Source.tick) of some time interval and use that within statefulMapConcat.

For a more indepth discussion of implementing sliding windows in Akka Streams I recommend reading this blog post from Software Mill.

2 Likes

Thanks for the reply Sean. It does indeed seem that merge + statefulMapConcat is the best option for me. I’d been put off using merge because the two streams have different types, but casting to Object before the merge, and downcasting in statefulMapConcat works. Wrapping this in a FanInShape2 makes it seem less ugly also.
The cardinality of the keys is quite low so for now I don’t have to deal with the time windowing, but this requirement will certainly come up sooner or later, so thanks for the info there too.
Cheers, Mark