Cinnamon plugin not reporting Streams and Persistence metrics

Akka Libraries Akka Streams & Alpakka
1

Hi! I searched everywhere in the docs and on the web for this problem but I have been unable to find an answer (or even an explicit error, the most frustrating part).

I set up the Cinnamon Agent using Grafana and Prometheus as the backend plugin following the docs. It, for the most part, works flawlessly. I’m using Cinnamon 2.20.1 and the corresponding Akka 2.9.3 version for all of Akka deps.

The issue is that, both in the sandbox environment as well as on my homelab K8S environment, pretty much none of the Akka Streams and Akka Persistence metrics work. Grafana dashboards are empty, and when checking the data Prometheus is scraping from Cinnamon at :9001/metrics I see that none of the metrics documented here are present in the API response of the Prometheus Cinnamon server.

So basically, the core issue is that even though I enabled both the Cinnamon.library.cinnamonAkkaPersistence and Cinnamon.library.cinnamonAkkaStream in my libraryDependencies (and I see some INFO logs regarding them, and no error ones) it seems that Cinnamon is not reporting their corresponding metrics.

Below I attach both the Grafana dashboard in case is of any help to anyone, and the initialization logs from Cinnamon. Any help is MASSIVELY appreciated!

https://grafana-fiubakka.marcosrolando.uk/dashboards
User: admin
Password: fiubakka

[info] running (fork) Main 
[info] [INFO] [06/18/2024 16:28:15.695] [main-1] [Cinnamon] Agent version 2.20.1
[info] [INFO] [06/18/2024 16:28:15.708] [main-1] [Cinnamon] Cinnamon Agent arguments: loglevel=INFO
[info] [INFO] [06/18/2024 16:28:15.774] [main-1] [Cinnamon] Agent found Scala Library version: 3.4.2
[info] [INFO] [06/18/2024 16:28:15.779] [main-1] [Cinnamon] Agent found Scala Futures version: 3.4.2
[info] [INFO] [06/18/2024 16:28:15.806] [main-1] [Cinnamon] Agent found Akka Actor version: 2.9.3
[info] [INFO] [06/18/2024 16:28:15.812] [main-1] [Cinnamon] Agent found Akka version: 2.9.3
[info] [INFO] [06/18/2024 16:28:15.820] [main-1] [Cinnamon] Agent found Akka Actor Typed version: 2.9.3
[info] [INFO] [06/18/2024 16:28:15.821] [main-1] [Cinnamon] Agent found Akka Persistence version: 2.9.3
[info] [INFO] [06/18/2024 16:28:15.823] [main-1] [Cinnamon] Agent found Akka Cluster version: 2.9.3
[info] [INFO] [06/18/2024 16:28:16.017] [main-1] [Cinnamon] Agent found Java Futures version: 11.0.21
[info] [INFO] [06/18/2024 16:28:16.322] [main-1] [Cinnamon] Agent found Akka Cluster version: 2.9.3
[info] [INFO] [06/18/2024 16:28:16.324] [main-1] [Cinnamon] Agent found Akka Remote version: 2.9.3
[info] [2024-06-18 16:28:16,593] [INFO] [akka.event.slf4j.Slf4jLogger] [] [fiubakka-server-akka.actor.default-dispatcher-4] - Slf4jLogger started MDC: {}
[info] [INFO] [06/18/2024 16:28:16.603] [main-1] [Cinnamon] Agent found Akka Stream version: 2.9.3
[info] [INFO] [06/18/2024 16:28:16.609] [main-1] [Cinnamon] Agent found Alpakka Kafka version: 6.0.0
[info] [2024-06-18 16:28:16,695] [INFO] [akka.remote.artery.ArteryTransport] [] [fiubakka-server-akka.actor.default-dispatcher-4] - Started embedded media driver in directory [/dev/shm/aeron-marcos/fiubakka-server-c94eefd5-27f4-4ce5-8e3d-b88ef1a2ca4f] MDC: {akkaAddress=akka://fiubakka-server, sourceThread=main, akkaTimestamp=19:28:16.693UTC, akkaSource=ArteryTransport(akka://fiubakka-server), sourceActorSystem=fiubakka-server}
[info] [2024-06-18 16:28:16,747] [INFO] [cinnamon.akka.stream.CinnamonAkkaStreamInstrumentation] [] [fiubakka-server-akka.actor.default-dispatcher-4] - Created MessageClassFormatter [cinnamon.akka.message-class-to-simple-name] MDC: {akkaAddress=akka://fiubakka-server, sourceThread=fiubakka-server-akka.actor.internal-dispatcher-2, akkaTimestamp=19:28:16.747UTC, akkaSource=CinnamonAkkaStreamInstrumentation, sourceActorSystem=fiubakka-server}
[info] [2024-06-18 16:28:16,756] [INFO] [cinnamon.akka.stream.CinnamonAkkaStreamInstrumentation] [] [fiubakka-server-akka.actor.default-dispatcher-4] - Created MessageClassFormatter [cinnamon.akka.message-class-to-simple-name] MDC: {akkaAddress=akka://fiubakka-server, sourceThread=fiubakka-server-akka.actor.internal-dispatcher-2, akkaTimestamp=19:28:16.756UTC, akkaSource=CinnamonAkkaStreamInstrumentation, sourceActorSystem=fiubakka-server}
[info] [2024-06-18 16:28:16,757] [INFO] [cinnamon.akka.stream.CinnamonAkkaStreamInstrumentation] [] [fiubakka-server-akka.actor.default-dispatcher-4] - Created MessageClassFormatter [cinnamon.akka.message-class-to-simple-name] MDC: {akkaAddress=akka://fiubakka-server, sourceThread=fiubakka-server-akka.actor.internal-dispatcher-2, akkaTimestamp=19:28:16.757UTC, akkaSource=CinnamonAkkaStreamInstrumentation, sourceActorSystem=fiubakka-server}
[info] [2024-06-18 16:28:16,834] [INFO] [akka.remote.artery.ArteryTransport] [] [fiubakka-server-akka.actor.default-dispatcher-4] - Remoting started with transport [Artery aeron-udp]; listening on address [akka://fiubakka-server@127.0.0.1:25520] with UID [2158047095399603059] MDC: {akkaAddress=akka://fiubakka-server@127.0.0.1:25520, akkaUid=2158047095399603059, sourceThread=main, akkaSource=ArteryTransport(akka://fiubakka-server), sourceActorSystem=fiubakka-server, akkaTimestamp=19:28:16.833UTC}
[info] [INFO] [06/18/2024 16:28:16.843] [main-1] [Cinnamon] Agent found Akka Cluster Sharding version: 2.9.3
[info] [INFO] [06/18/2024 16:28:16.844] [main-1] [Cinnamon] Agent found Akka Cluster Sharding Typed version: 2.9.3
[info] [INFO] [06/18/2024 16:28:16.845] [main-1] [Cinnamon] Agent found Akka Split Brain Resolver version: 2.9.3

type or paste code here

1 Like
2

For anyone that comes across this post, they were 2 separate issues in the end.

For Akka Streams, it turned out to be using “handleWith” (or wrappers of it), which is not instrumentable by Cinnamon. While its documented, its only done so if you are instrumenting via code (ie. instrument()), but I was instrumenting via Name selectors. It turns out that a Stream using “handleWith” is not instrumentable at all. You have to either instrument it by code using the given compatible function or rewrite the Stream (if possible, in my case it was) to a supported syntax.

Regarding Akka Persistence, it turned out to be that almost all metrics are only associated to Event Sourcing Persistence. Since I’m using Durable State, they don’t report anything. The only metrics that are also reported for Durable State are the Permits. It’s weird why some metrics don’t trigger with Durable State when I think they could (like Active Entities) but the other metrics don’t make sense in the Durable State paradigm, so its OK.