Akka Persistence Cassandra - Received Sequence Numbers out of order

Akka Persistence / Event Sourcing
1

Hello, form couple of days I am trying to deal with error I get from Akka Persistance Cassandra library:

[akka.tcp://affiliate-system@127.0.0.1:2551/system/cassandra-journal/tagWrites/all]
Received sequence numbers out of order. Pid: 0fc9bcf0-971f-45d6-962a-07db63901c46. 
HighestSeq: 7. New events: Vector((0fc9bcf0-9...

My setup is:

  • local machine
  • cassandra 3.11 in docker
  • keyspace created with settings WITH REPLICATION = {'class' : 'SimpleStrategy', 'replication_factor' : 1}
  • actor extends akka.persistence.PersistenceActor

I am migrating data from old system to new one, with persistenceId as UUID. I run the migration in a single thread, sending commands to persistent actor.
Actor is returning Future responses. I proceed only when future is done by calling andThen, or waiting manually for completion (with Await.result).

I am trying to figure out what can cause the problem. Do you have any ideas what else should I check, and where those incosistency can come from? I am not using AtLeastOnceDelivery trait, only simple actor sending messages with ask method akka.pattern.ask(actorRef, command).mapTo[T].

I will be very grateful for getting any help. I can provide you with more details if necessary.

Full stacktrace:

[ERROR] [08/31/2018 14:22:08.853] [affiliate-system-cassandra-plugin-default-dispatcher-52] [akka.tcp://affiliate-system@127.0.0.1:2551/system/cassandra-journal/tagWrites/all] Received sequence numbers out of order. Pid: 0fc9bcf0-971f-45d6-962a-07db63901c46. HighestSeq: 7. New events: Vector((0fc9bcf0-971f-45d6-962a-07db63901c46,7,7ad12f70-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:615)), (b89e255d-43ec-4c3f-a141-d911c61c563b,9,7ad415a0-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:634)), (d0e3a416-1954-4384-a8d2-788b5a582748,1,7ad686a0-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:650)), (b89e255d-43ec-4c3f-a141-d911c61c563b,10,7ad6adb0-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:651)), (67a6be7f-a0bb-49bb-a0ad-b6114e5bace5,1,7ad83450-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:661)), (b89e255d-43ec-4c3f-a141-d911c61c563b,11,7ad88270-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:663)), (b4c43594-a8e8-4933-bb25-8ec74907c17a,1,7ada7e40-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:676)), (b89e255d-43ec-4c3f-a141-d911c61c563b,12,7adacc60-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:678)), (68ce88ea-9d26-4ec8-ba3f-91cf896f54c7,1,7adc5300-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:688)), (b89e255d-43ec-4c3f-a141-d911c61c563b,13,7adcc830-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:691)), (29f6d727-6a70-4344-b9fc-b78dbfe5c55d,1,7adec400-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:704)), (b89e255d-43ec-4c3f-a141-d911c61c563b,14,7adf3930-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:707)), (e3cb7531-8b91-4420-84e3-9f957e671731,4,7ae1f850-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:725)), (b89e255d-43ec-4c3f-a141-d911c61c563b,15,7ae24670-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:727)), (4370492e-39c4-4e9b-917f-1f1fc32e9851,1,7ae41b30-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:739)), (b89e255d-43ec-4c3f-a141-d911c61c563b,16,7ae4de80-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:744)), (51c4366d-7efc-4f0b-8345-f4e71fb20a58,4,7ae9c080-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:776)), (b89e255d-43ec-4c3f-a141-d911c61c563b,17,7aea35b0-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:779)), (e1a32cde-fc1f-44d9-892e-26afbf815363,4,7aed6a00-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:800)), (b89e255d-43ec-4c3f-a141-d911c61c563b,18,7aeddf30-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:803)), (a583ddb6-298c-4c3e-9aba-e5e48137e9cd,4,7af13a90-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:825)), (b89e255d-43ec-4c3f-a141-d911c61c563b,19,7af1afc0-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:828)), (0268e1f1-4e09-4023-b6eb-c5dd9e5fee23,4,7af420c0-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:844))). Buffered events: Vector((0fc9bcf0-971f-45d6-962a-07db63901c46,7,7ad12f70-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:615)), (b89e255d-43ec-4c3f-a141-d911c61c563b,9,7ad415a0-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:634)), (d0e3a416-1954-4384-a8d2-788b5a582748,1,7ad686a0-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:650)), (b89e255d-43ec-4c3f-a141-d911c61c563b,10,7ad6adb0-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:651)), (67a6be7f-a0bb-49bb-a0ad-b6114e5bace5,1,7ad83450-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:661)), (b89e255d-43ec-4c3f-a141-d911c61c563b,11,7ad88270-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:663)), (b4c43594-a8e8-4933-bb25-8ec74907c17a,1,7ada7e40-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:676)), (b89e255d-43ec-4c3f-a141-d911c61c563b,12,7adacc60-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:678)), (68ce88ea-9d26-4ec8-ba3f-91cf896f54c7,1,7adc5300-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:688)), (b89e255d-43ec-4c3f-a141-d911c61c563b,13,7adcc830-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:691)), (29f6d727-6a70-4344-b9fc-b78dbfe5c55d,1,7adec400-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:704)), (b89e255d-43ec-4c3f-a141-d911c61c563b,14,7adf3930-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:707)), (e3cb7531-8b91-4420-84e3-9f957e671731,4,7ae1f850-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:725)), (b89e255d-43ec-4c3f-a141-d911c61c563b,15,7ae24670-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:727)), (4370492e-39c4-4e9b-917f-1f1fc32e9851,1,7ae41b30-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:739)), (b89e255d-43ec-4c3f-a141-d911c61c563b,16,7ae4de80-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:744)), (51c4366d-7efc-4f0b-8345-f4e71fb20a58,4,7ae9c080-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:776)), (b89e255d-43ec-4c3f-a141-d911c61c563b,17,7aea35b0-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:779)), (e1a32cde-fc1f-44d9-892e-26afbf815363,4,7aed6a00-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:800)), (b89e255d-43ec-4c3f-a141-d911c61c563b,18,7aeddf30-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:803)), (a583ddb6-298c-4c3e-9aba-e5e48137e9cd,4,7af13a90-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:825)), (b89e255d-43ec-4c3f-a141-d911c61c563b,19,7af1afc0-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:828)), (0268e1f1-4e09-4023-b6eb-c5dd9e5fee23,4,7af420c0-ad18-11e8-94da-1973a9be3626 (2018-08-31 12:22:08:844)))
[ERROR] [08/31/2018 14:22:08.857] [affiliate-system-cassandra-plugin-default-dispatcher-52] [akka://affiliate-system/system/cassandra-journal/tagWrites/all] Not received sequence numbers in order for pid: 0fc9bcf0-971f-45d6-962a-07db63901c46, current highest sequenceNr: 7. Sequence nr in event: 7
java.lang.IllegalStateException: Not received sequence numbers in order for pid: 0fc9bcf0-971f-45d6-962a-07db63901c46, current highest sequenceNr: 7. Sequence nr in event: 7
	at akka.persistence.cassandra.journal.TagWriter.$anonfun$validate$1(TagWriter.scala:124)
	at akka.persistence.cassandra.journal.TagWriter.$anonfun$validate$1$adapted(TagWriter.scala:113)
	at scala.collection.immutable.List.foreach(List.scala:389)
	at akka.persistence.cassandra.journal.TagWriter.validate(TagWriter.scala:113)
	at akka.persistence.cassandra.journal.TagWriter$$anonfun$akka$persistence$cassandra$journal$TagWriter$$idle$1.applyOrElse(TagWriter.scala:156)
	at akka.actor.Actor.aroundReceive(Actor.scala:517)
	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
	at akka.persistence.cassandra.journal.TagWriter.akka$actor$Timers$$super$aroundReceive(TagWriter.scala:84)
	at akka.actor.Timers.aroundReceive(Timers.scala:55)
	at akka.actor.Timers.aroundReceive$(Timers.scala:40)
	at akka.persistence.cassandra.journal.TagWriter.aroundReceive(TagWriter.scala:84)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588)
	at akka.actor.ActorCell.invoke(ActorCell.scala:557)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2

Ok, I found the root cause and solution :boom::boom::boom::boom:

So, here’s the story. I had two actors: ActorA and ActorB. They used user.uuid() as persistenceId.
It should work smoothly, having in mind that uuid are unique. The assumptions was that user can be either ActorA or ActorB. BUT, the assumption was wrong.

It occurred that user can be ActorA and ActorB at the same time. So for the same user both actors were using same persistenceId, which was user UUID.

To resolve that I just added prefix to each actor persistenceId, so they are not equal anymore.

// ActorA
override def persistenceId: String = s"actorA-$userUuid"

// ActorB
override def persistenceId: String = s"actorB-$userUuid"

And that’s it ;)

3

Thanks for sharing.

1 Like