What is the expected behavior for messages from a queue being processed by a Molecule. It appears that messages on a queue gain an affinity towards a specific node.
i.e. we have had a node go offline and the messages did not process until the node came back online. 3 scenarios about this concern me.
- Node offline – messages are simply not processed until the node comes back causing a delay in processing.
- VM is replaced - If the underlying VM has to be replaced and a node is added back into the Molecule, what happens to the messages ‘waiting’ to be processed?
- Scale-able Molecule – Through automation, horizontally scaling nodes. If nodes are instantiated and terminated, will messages get into a limbo state ‘waiting’ on the original node?