January 2017 Release Deep Dive: Amazon SQS Connector

Blog Post created by vreddy Employee on Jan 11, 2017

We're excited to announce the availability of the Amazon SQS Connector in the January 2017 release. Let's take a closer look at this connector and how to configure it within your integration processes.


Before we jump right in, here's a little context: Amazon Simple Queue Service (SQS) is a popular cloud-based message queuing service provided by the Amazon Web Services (AWS) platform. Amazon describes their SQS service as "...a fast, reliable, scalable, fully managed message queuing service. Amazon SQS makes it simple and cost-effective to decouple the components of a cloud application." The Amazon SQS connector now allows you to easily incorporate the Amazon queue service into your integration and data flow orchestrations. This connector complements the Amazon S3 connector to provide additional connectivity to the AWS platform using AtomSphere.


For complete connector documentation, please see Amazon SQS connector.

A typical queue connector provides the capability to send, receive/get and listen for messages on a queue or a topic. In this release, the SQS connector provides support for the Send and Receive actions. Listen support to receive and execute messages in real time will be added in a future release.

 Get the example process used below from the Process Library here.


How to Use


Create the AWS Queue

Create a queue in AWS. You can edit the default values if needed.



Configure the Connection Component

Set up the SQS connection in AtomSphere by choosing the appropriate AWS region and entering your AWS access key and secret key.



Send a Message to the Queue

Create a new Send SQS operation in AtomSphere. You can do this by creating a new operation and clicking Import in the operation.



Set up your process to send a test message using a Message shape to hard code a dummy document payload.



Populate the message shape with some data.



Run the process to send this message to the queue.


Receive a Message from the Queue

Create a new Receive SQS operation in AtomSphere. You can do this by creating a new operation and clicking Import in the operation.



In the above example we have configured the operation to receive one message with a visibility timeout of 120 seconds which means the message(s) will be "hidden" from other clients attempting to receive messages for two minutes.


By default the connector performs a "short poll" meaning it will connect, check for messages, and immediately disconnect even if no messages were found. However you can alternatively perform a "long poll" by configuring the Wait Timeout seconds to a value greater than 0 (up to 20). In this case the connector will connect but wait around for messages until the Wait Timeout or Maximum Message Count are reached.


Also note the Delete Messages setting. If checked, messages are removed from the queue immediately upon reading into the AtomSphere process. This means if those messages later fail during the execution, they can only be retried from the AtomSphere execution itself; there are no longer available to be received from the SQS queue again. Alternatively if you leave this setting unchecked, you should take care to go back and delete the messages as part of your process orchestration (see next section).


Deleting a Message from the Queue

Similarly you set up the Delete SQS operation in AtomSphere. As mentioned above, this action should be used if you decide to not delete messages immediately upon receiving, but rather explicitly delete them from the queue as part of your process orchestration upon successfully delivering to the destination system.



To delete a specific message, configure a Parameter on the SQS Send connector shape and set the delete request profile element "id" to Document Property > Connectors > Amazon SQS > Receipt Handle. This property is value is set automatically by the original Amazon SQS Receive operation.






Is it possible for me to get more than 10 documents at a time from the queue?

Amazon SQS allows you to receive up to 10 messages from a queue. Please refer to this article for additional information about this behavior.


Where can I find additional information about SQS?

This article hosted by Amazon provides additional information about SQS.