Can someone explain the scenarios/use-cases where Akka or Kafka are relevant? Can you give me some examples where Akka is preferred over Kafka or vice versa.
It’s hard to answer this question, because it’s so broad. And it’s very much comparing apples and oranges. It’s actually quite common to have both Akka and Kafka be part of a solution.
From the websites:
Akka is a toolkit for building highly concurrent, distributed, and resilient message-driven applications for Java and Scala
So, use cases for Akka are anytime you are building an application and you have either concurrency needs, distributed computing needs, or you need to build your application around async message passing. (Note, that in this context we are talking about messages passed internally between Akka actors.) A lot of applications are distributed these days, so I think that makes Akka pretty broadly useful.
But one thing you’ll notice is that Akka is primarily about building applications.
Apache Kafka® is a distributed streaming platform . What exactly does that mean?
A streaming platform has three key capabilities:
- Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system.
- Store streams of records in a fault-tolerant durable way.
- Process streams of records as they occur.
Kafka is typically described as a durable log. Kafka is primarily about data, streams of data to be specific. Kafka lists some of its common use cases as:
-
Messaging (note that in this case they are talking about a “replacement for a traditional message broker”. i.e. they are typically talking about messaging between systems, not within a system.)
-
Acitivity Tracking (i.e. storing the data about page views, searches, etc.)
-
Storing Metrics
-
Log Aggregation
Observe that nearly all of the use cases they list are primarily describing the data.
Thus it’s really common for an application to be built on Akka that read and writes data to Kafka. (Often by the Alpakka Kafka Connector.) I hate to oversimplify the question, but in some ways its like asking for use cases for when you would use Java versus use cases where you would a database. One is for applications, one is for data. While it’s true there is overlap (you can send messages in both Akka and Kafka, you can build logic in the application or you can build logic in stored procedures), but in both cases they generally are complementary not competitive.
P.S.
This question does become even more complicated by the fact that both Akka and Kafka are not so much technologies, but families of technologies. The above compares “core Akka” with “core Kafka”, but comparing say “Akka Streams” versus “Kakfa Streams” is more nuanced.
To provide some concrete example:
I use Kafka as a Stream Buffer to persist incoming data streams before any further processing. Sometimes I would also persist an “intermediate” stream in Kafka between two major processing stages, in case I am interested in this intermediate stream or need to provide it to several different downstream consumers.
I use Akka Streams as the technology to produce, consume and process data streams in various ways and use cases. In my opinion Akka Streams still has the best stream modelling (the Source, Flow, Sink model combined with a fluent API which create immutable and reusable stream processing recipes). Also it gives you resilience and back-pressure out of the box.
Note that I also I use Cassandra as database for time series. A time series can be thought of as a stream, but if you need to query for parts of one or more time series the Cassandra Query Language (CQL) and the way Cassandra stores and indexes data helps a lot, while in Kafka you are usually only able to stream a whole partition from some offset onwards. This is also the reason why for e. g. Event Sourcing many people choose something like Cassandra as the main Event Store and use Kafka as buffer or secondary stream storage.
Both Akka and Kafka are excellent technologies that really shine in production.
Thank you very much for your reply.
Thanks you very much for your reply.