Hi! I have been using Akka for my University final project and, as part of the project, I set up the Cinnamon plugin following the docs based on the Grafana + Prometheus setup. While it works for the most part, I’m having trouble making Cinnamon report both my Streams and Persistence metrics. I have both Cinnamon modules loaded as libraryDependencies (as the docs show) and the corresponding entries in the application.conf file, like so:
cinnamon {
akka {
actors {
"/user/*" {
report-by = instance
}
"sharded-group" {
report-by = instance
includes = ["/system/sharding/*"]
excludes = ["akka.cluster.sharding.Shard"]
}
}
streams {
"name:GameEventConsumer" {
report-by = name
}
"name:GameEventProducer" {
report-by = name
}
}
persistence.entities {
"sharded:?" {
report-by = group
}
}
cluster {
domain-events = on
member-events = on
node-status = on
singleton-events = on
shard-region-info = on
node-metrics = on
split-brain-resolver-events = on
}
ask-pattern.actors {
"akka.cluster.sharding.ShardRegion" {
report-by = instance
}
}
}
prometheus {
exporters += http-server
}
}
but the Grafana dashboards for both Akka Streams and Akka persistence are missing all data (the rest work perfectly fine). This happens both in my local environment and on my kubernetes deployment. Looking into it, I noticed the root cause is that the Cinnamon Prometheus HTTP server is not reporting any of the corresponding metrics for Streams and Persistence that both the docs state it should report, and that the dashboards consume.
I reread the docs like a dozen times carefully and for the life of me can’t figure out the root cause. Furthermore, Akka Cinnamon does not show any errors, the INFO logs show that it loaded the Streams instrumentation for example, but alas, no metrics for it are reported. I tried a bunch of different configurations in application.conf as well, to no avail.
Following are the initialization logs from the application. Any help is GREATLY appreciated!
[info] running (fork) Main
[info] [INFO] [06/19/2024 09:42:27.003] [main-1] [Cinnamon] Agent version 2.20.1
[info] [INFO] [06/19/2024 09:42:27.016] [main-1] [Cinnamon] Cinnamon Agent arguments: loglevel=INFO
[info] [INFO] [06/19/2024 09:42:27.082] [main-1] [Cinnamon] Agent found Scala Library version: 3.4.2
[info] [INFO] [06/19/2024 09:42:27.087] [main-1] [Cinnamon] Agent found Scala Futures version: 3.4.2
[info] [INFO] [06/19/2024 09:42:27.118] [main-1] [Cinnamon] Agent found Akka Actor version: 2.9.3
[info] [INFO] [06/19/2024 09:42:27.124] [main-1] [Cinnamon] Agent found Akka version: 2.9.3
[info] [INFO] [06/19/2024 09:42:27.133] [main-1] [Cinnamon] Agent found Akka Actor Typed version: 2.9.3
[info] [INFO] [06/19/2024 09:42:27.134] [main-1] [Cinnamon] Agent found Akka Persistence version: 2.9.3
[info] [INFO] [06/19/2024 09:42:27.136] [main-1] [Cinnamon] Agent found Akka Cluster version: 2.9.3
[info] [INFO] [06/19/2024 09:42:27.326] [main-1] [Cinnamon] Agent found Java Futures version: 11.0.21
[info] [INFO] [06/19/2024 09:42:27.637] [main-1] [Cinnamon] Agent found Akka Cluster version: 2.9.3
[info] [INFO] [06/19/2024 09:42:27.639] [main-1] [Cinnamon] Agent found Akka Remote version: 2.9.3
[info] [2024-06-19 09:42:27,903] [INFO] [akka.event.slf4j.Slf4jLogger] [] [fiubakka-server-akka.actor.default-dispatcher-4] - Slf4jLogger started MDC: {}
[info] [INFO] [06/19/2024 09:42:27.911] [main-1] [Cinnamon] Agent found Akka Stream version: 2.9.3
[info] [INFO] [06/19/2024 09:42:27.915] [main-1] [Cinnamon] Agent found Alpakka Kafka version: 6.0.0
[info] [2024-06-19 09:42:27,999] [INFO] [akka.remote.artery.ArteryTransport] [] [fiubakka-server-akka.actor.default-dispatcher-4] - Started embedded media driver in directory [/dev/shm/aeron-marcos/fiubakka-server-eccf2226-b67d-4449-a2af-2cb3a153fee8] MDC: {akkaAddress=akka://fiubakka-server, sourceThread=main, akkaTimestamp=12:42:27.998UTC, akkaSource=ArteryTransport(akka://fiubakka-server), sourceActorSystem=fiubakka-server}
[info] [2024-06-19 09:42:28,071] [INFO] [cinnamon.akka.stream.CinnamonAkkaStreamInstrumentation] [] [fiubakka-server-akka.actor.default-dispatcher-4] - Created MessageClassFormatter [cinnamon.akka.message-class-to-simple-name] MDC: {akkaAddress=akka://fiubakka-server, sourceThread=fiubakka-server-akka.actor.internal-dispatcher-2, akkaTimestamp=12:42:28.071UTC, akkaSource=CinnamonAkkaStreamInstrumentation, sourceActorSystem=fiubakka-server}
[info] [2024-06-19 09:42:28,080] [INFO] [cinnamon.akka.stream.CinnamonAkkaStreamInstrumentation] [] [fiubakka-server-akka.actor.default-dispatcher-4] - Created MessageClassFormatter [cinnamon.akka.message-class-to-simple-name] MDC: {akkaAddress=akka://fiubakka-server, sourceThread=fiubakka-server-akka.actor.internal-dispatcher-2, akkaTimestamp=12:42:28.080UTC, akkaSource=CinnamonAkkaStreamInstrumentation, sourceActorSystem=fiubakka-server}
[info] [2024-06-19 09:42:28,082] [INFO] [cinnamon.akka.stream.CinnamonAkkaStreamInstrumentation] [] [fiubakka-server-akka.actor.default-dispatcher-4] - Created MessageClassFormatter [cinnamon.akka.message-class-to-simple-name] MDC: {akkaAddress=akka://fiubakka-server, sourceThread=fiubakka-server-akka.actor.internal-dispatcher-2, akkaTimestamp=12:42:28.082UTC, akkaSource=CinnamonAkkaStreamInstrumentation, sourceActorSystem=fiubakka-server}
[info] [2024-06-19 09:42:28,163] [INFO] [akka.remote.artery.ArteryTransport] [] [fiubakka-server-akka.actor.default-dispatcher-4] - Remoting started with transport [Artery aeron-udp]; listening on address [akka://fiubakka-server@127.0.0.1:25520] with UID [7446274637274230606] MDC: {akkaAddress=akka://fiubakka-server@127.0.0.1:25520, akkaUid=7446274637274230606, sourceThread=main, akkaSource=ArteryTransport(akka://fiubakka-server), sourceActorSystem=fiubakka-server, akkaTimestamp=12:42:28.163UTC}
[info] [INFO] [06/19/2024 09:42:28.174] [main-1] [Cinnamon] Agent found Akka Cluster Sharding version: 2.9.3
[info] [INFO] [06/19/2024 09:42:28.175] [main-1] [Cinnamon] Agent found Akka Cluster Sharding Typed version: 2.9.3
[info] [INFO] [06/19/2024 09:42:28.176] [main-1] [Cinnamon] Agent found Akka Split Brain Resolver version: 2.9.3
[info] [2024-06-19 09:42:28,178] [INFO] [akka.cluster.Cluster] [] [fiubakka-server-akka.actor.default-dispatcher-4] - Cluster Node [akka://fiubakka-server@127.0.0.1:25520] - Starting up, Akka version [2.9.3] ... MDC: {akkaAddress=akka://fiubakka-server@127.0.0.1:25520, akkaUid=7446274637274230606, sourceThread=main, akkaSource=Cluster(akka://fiubakka-server), sourceActorSystem=fiubakka-server, akkaTimestamp=12:42:28.178UTC}
[info] [2024-06-19 09:42:28,209] [INFO] [akka.cluster.Cluster] [] [fiubakka-server-akka.actor.default-dispatcher-4] - Cluster Node [akka://fiubakka-server@127.0.0.1:25520] - Registered cluster JMX MBean [akka:type=Cluster] MDC: {akkaAddress=akka://fiubakka-server@127.0.0.1:25520, akkaUid=7446274637274230606, sourceThread=main, akkaSource=Cluster(akka://fiubakka-server), sourceActorSystem=fiubakka-server, akkaTimestamp=12:42:28.209UTC}
[info] [2024-06-19 09:42:28,210] [INFO] [akka.cluster.Cluster] [] [fiubakka-server-akka.actor.default-dispatcher-4] - Cluster Node [akka://fiubakka-server@127.0.0.1:25520] - Started up successfully MDC: {akkaAddress=akka://fiubakka-server@127.0.0.1:25520, akkaUid=7446274637274230606, sourceThread=main, akkaSource=Cluster(akka://fiubakka-server), sourceActorSystem=fiubakka-server, akkaTimestamp=12:42:28.209UTC}
[info] [2024-06-19 09:42:28,234] [INFO] [akka.cluster.sbr.SplitBrainResolver] [] [fiubakka-server-akka.actor.default-dispatcher-4] - SBR started. Config: strategy [KeepMajority], stable-after [20 seconds], down-all-when-unstable [15 seconds], selfUniqueAddress [akka://fiubakka-server@127.0.0.1:25520#7446274637274230606], selfDc [default]. MDC: {akkaAddress=akka://fiubakka-server@127.0.0.1:25520, akkaUid=7446274637274230606, sourceThread=fiubakka-server-akka.actor.default-dispatcher-19, akkaSource=akka://fiubakka-server/system/cluster/core/daemon/downingProvider, sourceActorSystem=fiubakka-server, akkaTimestamp=12:42:28.234UTC}
[info] [2024-06-19 09:42:28,237] [INFO] [akka.cluster.Cluster] [akkaMemberChanged] [fiubakka-server-akka.actor.default-dispatcher-19] - Cluster Node [akka://fiubakka-server@127.0.0.1:25520] - Node [akka://fiubakka-server@127.0.0.1:25520] is JOINING itself (with roles [dc-default], version [0.0.0]) and forming new cluster MDC: {akkaRemoteAddressUid=7446274637274230606, akkaAddress=akka://fiubakka-server@127.0.0.1:25520, akkaUid=7446274637274230606, sourceThread=fiubakka-server-akka.actor.internal-dispatcher-17, akkaSource=Cluster(akka://fiubakka-server), sourceActorSystem=fiubakka-server, akkaRemoteAddress=akka://fiubakka-server@127.0.0.1:25520, akkaTimestamp=12:42:28.236UTC, akkaMemberStatus=Joining}
[info] [2024-06-19 09:42:28,238] [INFO] [akka.cluster.Cluster] [] [fiubakka-server-akka.actor.default-dispatcher-19] - Cluster Node [akka://fiubakka-server@127.0.0.1:25520] - is the new leader among reachable nodes (more leaders may exist) MDC: {akkaAddress=akka://fiubakka-server@127.0.0.1:25520, akkaUid=7446274637274230606, sourceThread=fiubakka-server-akka.actor.internal-dispatcher-17, akkaSource=Cluster(akka://fiubakka-server), sourceActorSystem=fiubakka-server, akkaTimestamp=12:42:28.238UTC}
[info] [2024-06-19 09:42:28,243] [INFO] [akka.cluster.Cluster] [akkaMemberChanged] [fiubakka-server-akka.actor.default-dispatcher-19] - Cluster Node [akka://fiubakka-server@127.0.0.1:25520] - Leader is moving node [akka://fiubakka-server@127.0.0.1:25520] to [Up] MDC: {akkaRemoteAddressUid=7446274637274230606, akkaAddress=akka://fiubakka-server@127.0.0.1:25520, akkaUid=7446274637274230606, sourceThread=fiubakka-server-akka.actor.internal-dispatcher-17, akkaSource=Cluster(akka://fiubakka-server), sourceActorSystem=fiubakka-server, akkaRemoteAddress=akka://fiubakka-server@127.0.0.1:25520, akkaTimestamp=12:42:28.243UTC, akkaMemberStatus=Up}
[info] [2024-06-19 09:42:28,260] [INFO] [akka.cluster.sbr.SplitBrainResolver] [] [fiubakka-server-akka.actor.default-dispatcher-4] - This node is now the leader responsible for taking SBR decisions among the reachable nodes (more leaders may exist). MDC: {akkaAddress=akka://fiubakka-server@127.0.0.1:25520, akkaUid=7446274637274230606, sourceThread=fiubakka-server-akka.actor.default-dispatcher-19, akkaSource=akka://fiubakka-server/system/cluster/core/daemon/downingProvider, sourceActorSystem=fiubakka-server, akkaTimestamp=12:42:28.260UTC}
[info] [INFO] [06/19/2024 09:42:28.396] [main-1] [Cinnamon] Agent found Akka Persistence Typed version: 2.9.3
[info] [INFO] [06/19/2024 09:42:28.419] [main-1] [Cinnamon] Agent found Akka Cluster Tools version: 2.9.3