Molecule has a warning sign on it, however each and every node is online and healthy. For this example, we refer to a node with address id 1_1_1_3.
Example entries in container log:
[CONTAINER_VERSION_MISMATCH] Found different container version XXX in node.1_1_1_3.dat
[CLUSTER_STATE_MISMATCH] Found different cluster state XXX in node.1_1_1_3.dat
Clustering issues could cause this issue. One of the molecule node files (in this case node 1_1_1_3) did not get deleted properly when it was having issues.
- On the fileshare where the molecule is installed, under the molecule installation directory, go to the .../bin/views folder and look for the the number of node.XX.dat files to ensure one exists for each active node. Check to see if the file node.1_1_1_3.dat exists.
- If the file node.1_1_1_3.dat exists, compare the container version with the version of the other nodes to see if it is different from the other nodes. If it is different from the other nodes, proceed to step 3.
- In AtomSphere, go to Manage -> Atom Management and select the "Cluster Status" tab. Check the number of nodes and see if node 1_1_1_3 is there.
Delete the node.1_1_1_3.dat file from the .../bin/views folder and perform a rolling restart of the molecule nodes to see if it will clear up the clustering problem.
Longer term, clustering health should be monitored more closely for underlying issues between the nodes and themselves or between the nodes and the fileshare, especially if the situation becomes more frequent.