CONTAINER_VERSION_MISMATCH in Molecule container logs

Document created by sheng_liao462475 Employee on Oct 26, 2016Last modified by Adam Arrowsmith on Nov 13, 2016
Version 4Show Document
  • View in full screen mode

Issue

Molecule has a warning sign on it, however each and every node is online and healthy.  For this example, we refer to a node with address id 1_1_1_3.

 

Example entries in container log:

[CONTAINER_VERSION_MISMATCH] Found different container version XXX in node.1_1_1_3.dat
[CLUSTER_STATE_MISMATCH] Found different cluster state XXX in node.1_1_1_3.dat

 

Cause

Clustering issues could cause this issue. One of the molecule node files (in this case node 1_1_1_3) did not get deleted properly when it was having issues.

 

Troubleshooting Steps

  1. On the fileshare where the molecule is installed, under the molecule installation directory, go to the .../bin/views folder and look for the the number of node.XX.dat files to ensure one exists for each active node.  Check to see if the file node.1_1_1_3.dat exists.
  2. If the file node.1_1_1_3.dat exists, compare the container version with the version of the other nodes to see if it is different from the other nodes.  If it is different from the other nodes, proceed to step 3.
  3. In AtomSphere, go to Manage -> Atom Management and select the "Cluster Status" tab.  Check the number of nodes and see if node 1_1_1_3 is there.

 

 

Solution

Delete the node.1_1_1_3.dat file from the .../bin/views folder and perform a rolling restart of the molecule nodes to see if it will clear up the clustering problem.

 

Longer term, clustering health should be monitored more closely for underlying issues between the nodes and themselves or between the nodes and the fileshare, especially if the situation becomes more frequent.

1 person found this helpful

Attachments

    Outcomes