Molecule Down Procedures (also applies to Local Clouds)

Document created by mike_aronson Employee on Jan 30, 2014Last modified by mike_aronson Employee on Nov 21, 2016
Version 3Show Document
  • View in full screen mode
Your molecule or local cloud has gone down.  This could mean

 

  • Molecule nodes are not running on the server

  • Molecule went offline (is red in the platform), but nodes are still running on the server

  • Molecule did not restart after release control was applied

  • Molecule did not restart after a release

  • Molecule went down, but came back up and interested in root cause

  • Molecule not communicating properly with endpoints, platform or among nodes

 

This article covers steps that you can take to diagnose the issue, restore operations, and determine root cause to reduce/prevent it from happening in the future. 

 

 

1. Identify the problem:

  • Molecule nodes are not running on the server

  • Molecule went offline (is red in the platform), but nodes are still running on the server

  • Molecule did not restart after release control was applied

  • Molecule did not restart after a release

  • Molecule went down, but came back up and interested in root cause

  • Molecule not communicating properly with endpoints, platform or among nodes

2. Check Molecule node state of health:

To determine if molecule nodes are healthy, check molecule node state of health per the instructions at this link: http://help.boomi.com/atomsphere/GUID-1D2F16DA-E27E-42ED-B9A9-227E0139DA8F.html

  • Review the contents of the <installation_directory>\bin\views directory.

  • If one node is healthy, determine if the molecule can be sustained in this state.
     

3. Collect DIagnostic information:

  • Obtain the container logs from the <molecule installation_directory>\logs directory. There should be at least two container.log files (one for each node in the Molecule)

    • Review and capture/save the logs to see if there are any errors occurring during the time period when the molecule went down.

    • If the molecule went down during a release control attempt or during an actual release, the pending updates for release control would first download the new release files, then unzip them and then apply them.  You may be able to tell from the logs which step is failing.  Sometimes the release download is large and therefore the server or network may not be accepting the large file or timing out.

  • Review and capture/save the contents of the <installation_directory>\bin\views directory.
  • Review and capture/save the Restart.log file from the <installation_directory>\bin directory for errors

  • Review and capture/save the Restart.bat file from the <installation_directory>\bin directory for errors

  • Review and capture/save the Error.log file from the <installation_directory>\bin directory for errors

  • Reivew and capture/save any hs_pid_err_xxxx.log files that exist with same date from the <installation_directory> and review the file for errors.

4. Take Corrective Actions:

After changes are made to the Dell Boomi Atom, the Atom (or Molecule or Atom Cloud) must restart itself before the changes can take effect. Your Atom will not restart itself while integration processes are running. Instead it waits for the processes to finish. It does not start any more processes while it waits. If a process that takes hours to complete is running, the Atom will have to wait a long time before it can restart. You can specify the number of minutes that AtomSphere will wait before it forces the Atom to restart. Any processes that are currently running will be aborted. However, it will not abort processes that are currently running in a Molecule or Atom Cloud that uses forked execution.

  • Attempt to restart the atom/molecule/cloud.  Recommend stopping the nodes, waiting 5 minutes, then start the nodes one at a time.  Check the node View files (#2 above) to diagnose/confirm if the nodes are communicating again: http://help.boomi.com/atomsphere/GUID-1860A772-3EB2-4DBE-AB6C-54C6CF854294.html

  • If the attempts to restart the atom/molecule/cloud are unsuccessful, open a support case (#5 below) and obtain and attach the container log from the <atom installation folder>/logs folder and the contents of the <installation_directory>\bin\views folder to the case.  We will need these files at a minimum in order to diagnose the issue. Then attempt to reboot the servers hosting atom/molecule/cloud nodes and then try the step above again.

  • If communications issues are still occurring, attempt to refresh any proxies if they are configured for the molecule/atom/cloud:

http://help.boomi.com/atomsphere/GUID-20C748C6-05EA-41FF-8750-F47D4873FA9D.html


5. Open a Support Case:

If you are unable to restart your molecule or would like assistance in diagnosing root cause, in AtomSphere, select Help & Feedback -> Support & Community and open a case via the Dell Boomi Support portal.  Include the following information in the case:

  • A Phone Number and email at which you can be reached

  • The Molecule name from Manage -> Atom Management

  • A detailed description of the issue and desired request (help with restarting molecule, request for root cause analysis, etc…)

  • Steps that have been taken already

  • Contents of the <installation_directory>\bin\views directory.

  • Attach the container logs from the <installation_directory>\logs directory. There should be at least two container.log files (one for each node in the Molecule)

  • Attach the Restart.log file from the <installation_directory>\bin directory

  • Attach the Restart.bat file from the <installation_directory>\bin directory

  • Attach the Error.log file from the <installation_directory>\bin directory

  • Attach the hs_pid_err_xxx.log file if it exists with same date in <installation_directory>/bin

  • If it is a release control issue and the connector version is not updating in the Atom Management UI, obtain the connector files from the <installation_directory>/connector/connectorname folder

  • atom.vmoptions file from <installation_directory>/bin

  • container.properties files from <installation_directory>/conf

  • start.properties file from <installation_directory>/conf

4 people found this helpful

Attachments

    Outcomes