More than one singleton running in a cluster

Hi,

I have a 4 nodes cluster running zato-2.0.7. I noticed that sometime, in the logs/singleton.log, more than 1 singleton were running as it shows “Cluster-wide singleton keep-alive OK, serverid:[1]” twice with 2 different server id. I know a cluster should have only one singleton running, so i shutdown one and restart it but when it came back up, it still vote itself as singleton node. I have to shutdown the entire cluster and bring up one by one in order to resolve the issues.

Anyone encounter that before? Will the cluster resolve this by itself when it notice more than 1 singleton running?

Is there a way to resolve this without shutting down the entire cluster as we are going to put the cluster to production soon.

The cluster is running with redis-3.2 and mysql-5.6

Thanks.

Example of logs showing:

2016-10-06 17:22:39,896 - INFO - 21259:Dummy-4196 - zato_singleton:22 - Cluster-wide singleton keep-alive OK, server id:[2], name:[server1-ext1]
2016-10-06 17:23:38,389 - INFO - 21259:Dummy-4201 - zato_singleton:22 - Cluster-wide singleton keep-alive OK, server id:[4], name:[server2-ext2]
2016-10-06 17:24:09,903 - INFO - 21259:Dummy-4204 - zato_singleton:22 - Cluster-wide singleton keep-alive OK, server id:[2], name:[server1-ext1]
2016-10-06 17:26:08,416 - INFO - 21258:Dummy-6422 - zato_singleton:22 - Cluster-wide singleton keep-alive OK, server id:[4], name:[server2-ext2]

Keith

Anyone can help? Or can someone point to me which key/table does singleton update it’s flag in redis/mysql?

Thanks

Keith