Combining materialized values in Java GraphDSL.Builder

Hi Gergő, thanks for your patience with me :slight_smile: and I don’t say the toolkit has missing parts. Most likely it is my limited knowledge about the toolkit and even my concept might be wrong.

To clarify what I am trying to achieve, I have created another sketch. In my system the user can configure an arbitrary workflow, that consists of a single input source which delivers ByteStrings. A special Flow converts these ByteString to Messages (you can think of them like something similar to a XML DOM Document). These messages are then processed within the workflow. Filtered, routed, enriched etc. In the end the documents are then written to one or more Sinks which are converting the Messages into Files, Kafka Messages or whatever. I mapped that workflow to the following Akka Stream components:

If one of my workflow nodes has more than one input or output, I map it into a combination of FanIn -> Flow -> FanOut. If the number of inputs or outputs are one, the FanIn and FanOut stages are suppressed.

As said previously, all that is already working perfectly. I was only struggeling with the problem, that I would like to get the materialized values of

a) the single Flow<ByteString,Message>
b) all the Flow<Message,Message>
c) all the Sink<Message>

stages.

I have splitted my graph at the boundary Source<ByteString> => Flow<ByteString,Message>. At that splitting point, my two graph components are “real” Akka Streams processing stages. The left side is the Source<ByteString> while the right side is a Sink<ByteString>.

To create the Sink on the right side, I tried to use your new API. As far as I understood I can add a List<> of a certain Graph type to the builder. In my case I have two of these lists:

a) a List<Sink<Message>>
b) a List<Flow<Message,Message>>

But in both cases the result of that GraphDSL.create(List<...> ...) call is neither aSink<Message> (with a single input) nor a Flow<Message,Message> with a single input and output. In both cases the result is an “OpenShape” with an arbitrary number of inputs and outputs (for theList<Flow> case) and a “SinkShape” with an arbitrary number of inputs for the List<Sink> case.

What I am missing (or what I don’t know about) is the chance to return an arbitrary Graph shape form the GraphDSL.create() method which is not necessary a strict Flow or Sink with a single input/output but instead allows an arbitrary number of open input/outputs. I will look into your suggestion of creating my own shape.

I hope this clarifies what I am trying to achieve. Any comments and suggestions are highly welcome.

Regards,
Lay