Why gossip send full state

Akka Libraries Akka Cluster
1

Hello, i faced with problem that each node in two nodes cluster send full state of two ORMap every
distributed-data.gossip-interval.

I used akka 2.5.13, artery for transport, two actors (on each node), each actor send to replicator own changes of ORMap<String, Dgroup>.

Dgroup that type extends AbstractReplicatedData and implements ReplicatedDataSerialization.

Here’s the log

a.r.a.Encoder - sending remote message [Gossip(Map(event.3792209 → DataEnvelope(ORMap(
group.2043064276264301 → Dgroup{id=2043064276264301, factValues=PNCounterMap(), version=GCounter(0), factsSet=GSet()},
group.2041891226523599 → Dgroup{id=2041891226523599, factValues=PNCounterMap(), version=GCounter(0), factsSet=GSet()},
group.2041373232281131 ->Dgroup{id=2041373232281131, factValues=PNCounterMap(), version=GCounter(0), factsSet=GSet()},
group.2042184502568969 → Dgroup{id=2042184502568969, factValues=PNCounterMap(), version=GCounter(0), factsSet=GSet()}),Map(), VersionVector(UniqueAddress(akka://Test@127.0.0.1:2551,-663867923909857583) → 1, UniqueAddress(akka://Test@127.0.0.1:2552,5656432729175541792) → 1)),
event.1524823 → DataEnvelope(ORMap(
group.209142229049073 → Dgroup{id=209142229049073, factValues=PNCounterMap(), version=GCounter(0), factsSet=GSet()}),Map(),
VersionVector(UniqueAddress(akka://Test@127.0.0.1:2551,-663867923909857583) → 1, UniqueAddress(akka://Test@127.0.0.1:2552,5656432729175541792) → 1))),true)]
to [Actor[akka://Test@127.0.0.1:2552/system/ddataReplicator#55107212]] from [Actor[akka://Test/system/ddataReplicator#1630010406]]

This ORMap doesn’t change in my test but it still send full state.
As i know ORMap should be replicated by delta or full state depends on max-delta-elements .

What wrong with it ?

2

It’s right that it shouldn’t send full state gossip when the state is in sync. It’s important that the serializer produce the same bytes for equivalent content, because it’s a sha1 hash of the bytes that is sent in the Status messages to detect diverging content and need for full gossip. E.g. Elements in a Set must be sorted deterministically in the serializer.

If you can share your dgroup and serializer I could take a look and see if I spot anything that is wrong.

3

Thank you Patrik the StatusMessage hash very helpfull i will check it.

I’ve just reproduced other strange thing , i’ve created a map of Gcounters with keys a,b,c .
I don’t set any special serializers in config all by default.

ORMap<String, GCounter>

Why it sends full state of ORMap if i increment only one counter ?

Gossip(Map(mapkey -> DataEnvelope(ORMap(a -> GCounter(5), b -> GCounter(5), c -> GCounter(7)),Map(),VersionVector(UniqueAddress(akka://Test@127.0.0.1:2551,2140052748990380887) -> 9, UniqueAddress(akka://Test@127.0.0.1:2552,-3068087603267994073) -> 8))),true)

I update ORMap in this way

Update<ORMap<String, GCounter>> upd = new Update<>(
mapDataKey,
ORMap.create(),
Replicator.writeLocal(),
map -> map.updated(node, key, GCounter.create(), (gCounter -> gCounter.increment(node, 1L))));

I caught it , is it sends full state because hash changed ? , what does mean false - in Status message

ActorSelectionMessage(Status(mapkey → 756296aa7c54a70ebe7edbff60144ddec66820c6),Vector(system, ddataReplicator),false)

Is it possible to avoid sending full state of ORMap when some entries of map updated ?

Can you suggest the right way to implement replication process w/o sending full state of Map structure ?

4

For map of counters you should use PNCounterMap. I think it should be able to send delta of your case also. We improved something related in Akka 2.5.14 so give that a try.

Note that deltas are only a best effort optimization. If deltas are lost (e.g. during network partition) or when when new nodes are added it falls back to sending full state. For the normal updates it should be able to sync with deltas and not have to send full state.

If you still see a problem I suggest that you create an issue describing how to reproduce.