Functional Monitoring of Integration Processes Best Practices


This article provides an overview of the options and best practices for monitoring your integration processes.

 

 

Overview

Knowing when integration processes fail or do not complete as expected is a critical part of managing the integration platform. Boomi Enterprise Platform provides a number of options to suit your needs depending upon the volume of executions, desired communication method, and sophistication of notification routing and delivery requirements.

 

For monitoring the general availability and health of your basic runtimes and runtime clusters, see Operational Monitoring Techniques for basic runtime, runtime cluster, and runtime cloud Runtimes.

 

Functional Monitoring Options

The table below describes the various process monitoring options and best practices for use.

 

OptionDescriptionBest Practices

Process Reporting

(UI)

The Process Reporting console provides detailed information about process executions including process logs and documents.

 

Manage > Process Reporting

The Process Reporting console can be used to manually monitor low volumes of executions but as activity increases, handling Events on an exception-basis becomes necessary. However regardless of how you are notified about an error, you will most likely use the Process Reporting console to research logs and document-level information, troubleshoot, and rerun documents. See also Overview of Boomi Enterprise Platform Logs for Troubleshooting.

 

Use Document Tracking to capture key values from document data to be able to quickly find and trace a given record through multiple process executions.

 

Important notes about Process Reporting:

  • Results are not retained indefinitely. See the section below: A Note about Execution History Retention and Purge Policies.
  • Access to execution results can be restricted by user Role and/or Environment. Additionally the ability to view document data can be permissioned separately. See Understanding User Management and Custom Roles.
  • Executions for web service processes configured as Low Latency are NOT recorded in Process Reporting unless they have an exception. Instead they are summarized in the Real-time Dashboard (see below).

Dashboards

(UI)

The Dashboards provide historical and summary information about process executions and inbound requests.

 

Dashboard > choose dashboard

There are three dashboards that can be used for trending and historical analysis:

  • Account Dashboard - Account-level summary of activity including number of executions/errors, throughput, document count, and connector usage.
  • Real-time Dashboard (available if Services Enablement feature is enabled in your account) - Summary of published web services processes configured as Low Latency which do not capture execution history in Process Reporting.
  • HTTP Status Dashboard (available if Services Enablement feature is enabled in your account) - Summary of HTTP status code responses for published web services processes by date.

Email Alerts

(Event Framework)

Boomi Enterprise Platform users can choose to receive platform Events via email. Users can choose to subscribe to Events by Log Level and Type.

 

Account menu > Setup > Email Alerts

Email alerts are best used for exception-based notifications that will be received and acted upon by individuals. As a general recommendation, configure the Log Level to WARNING to receive alerts for WARNING and ERROR events but not expected successful activity such as "process started" and "process completed". See Email alert management.

 

For higher executions volumes or if more advanced subscription/routing rules are desired, use the Event API to consume and handle Events as needed. See the Platform API section below.

 

Important notes about email alerts:

  • Alert subscriptions are configured per user, not per account. Users with access to multiple accounts will see subscriptions across their accounts.
  • Administrators cannot subscribe users to emails on their behalf. This is notable when wanting to send emails to a distribution list or shared inbox. A separate Boomi Enterprise Platform user must be created. See: Setting up email distribution list for process alerts
  • Email alerts are batched and sent from the Boomi Enterprise Platform periodically every few minutes to avoid triggering spam filters. This means one email can contain multiple Events from different processes, basic runtimes, event types, log levels, etc.
  • The subject and format of the email messages cannot be modified.
  • Email alerts are NOT sent for Events occurring in "Test" Environments (i.e. Environments classified as "Test". However are generated and may be consumed through another channel such as API or RSS.
  • For Low Latency executions, process.execution Events are NOT generated, even for process errors and Exception steps, so no email alerts will be delivered. user.notification Events ARE created and therefore can be delivered via email, however be cautious in high volume scenarios. See the section below for considerations. See Understanding the Event Framework for more about Event Types.
  • Email alerts do not consume a connection license.

RSS Feeds

(Event Framework)

Platform Events are available via RSS feed per basic runtime or at the account level.

 

Account menu > Setup > Account Information

or

Manage > Runtime Management > choose basic runtime > basic runtime Information

RSS are the simplest approach for receiving platform Events. If you have an incident reporting system or monitoring tool that can consume RSS feeds, consider using them.

 

For exception-based monitoring, use the "Alerts Only" feeds to only receive Warning and Error level Events. There are no other filtering options.

 

See Understanding the Event Framework.

Platform API

(Event Framework)

Events can be consumed programmatically via the Event API.

 

There is no UI for the list of Events.

The Event object provides the most flexibility for routing rules, message formatting, and delivery options. It is the best option for exception-based monitoring of larger numbers of executions.

 

For execution-related Events, you can use the Execution Record object and Download process log operation to retrieve additional details about the execution. Note the API does not provide access to document-level information including tracked fields.

 

You can create another Boomi Enterprise Platform process (or use an external monitoring tool) to periodically extract new Events and route and deliver as required. For example, Events could be sent to your incident reporting application to leverage existing communication channels and escalation rules. Note the connection to the destination system including your company's SMTP mail server will consume a connection license.

 

IMPORTANT: Event records are available for 7 days.

 

See Understanding the Event Framework.

Custom

(Built into Process Flow)

Custom error handling and notification logic can be built into the process design/flow as required.

If truly custom notification logic is required, you can design this into the process flow itself by capturing errors and validation warnings using Try/Catch and Business Rules steps and connector operation response handling. For example, runtime errors could be caught, formatted with a Message step, and emailed directly to recipients using a Mail connector and your company's SMTP mail server.

 

Alternatively, the errors could be mapped to custom alert message XML format, and written to an external database or queue for subsequent processing. This option may be desired when integrating with an existing notification/auditing/monitoring framework or to provide custom attributes in the notification message. It requires additional upfront design, development, and testing effort. It is still advised to use the platform Event subscriptions to be notified of problems that cannot be handled within the context of a process execution.

 

A Note about Execution History Retention and Purge Policies

The Process Reporting console displays execution results for the past 30 days. This period cannot be changed. The results include process- and document-level metadata such as execution statistics, document counts, and document tracked fields.

 

The Process Reporting console will always retain the summary execution history for 30 days, however the execution artifacts such as process logs and document data can be purged more frequently. This may be desirable for basic runtimes/runtimes processing a large volume of data. Note that purge schedules are configured for the entire runtime, not per-process.


The runtime can also be configured to "Purge Immediately" so that execution artifacts (most notably document data) are removed immediately upon completion of the execution. Again keep in mind the Purge Immediately setting applies to the entire runtime, not individual processes.


Execution artifacts for processes running on the Boomi runtime cloud are purged after 14 days. This can be decreased but not increased beyond 14 days.


While it is technically possible to increase the purge history beyond 30 days for local runtimes, it is not practical because the execution summary information in the Process Reporting console is only available for 30 days.


The purge frequency can be modified per runtime in Runtime Management > basic runtime Properties (or runtime cluster or Cloud Properties). The Basic Property sets a “global” purge frequency for all execution artifacts. Additionally/alternatively you can override this setting by configuring purge frequencies for logs (process and runtime), documents (also applies to basic runtime Queues), and temp data individually with Advanced Properties.

 

For more information see Purging of basic runtime, runtime cluster, or runtime cloud logs and data.

 

Considerations for Monitoring High Volume Low Latency Processes

Processes configured to use Low Latency mode require special considerations for monitoring. As noted above, Low Latency process executions do not generate Events for process errors and therefore you cannot rely on email alerts, RSS feeds, or even the Event API. User notifications (from Notify steps) are created and therefore could be received via email alerts, however because Low Latency mode is typically used for high volume scenarios such as inbound web services and JMS messages, receiving an Event or email alert for each individual error can be impractical given the sheer volume of potential messages. Instead the strategy is to detect a general, persistent issue with the service and then use the available process logs and dashboards to investigate further.

 

Below are some additional considerations and recommendations for monitoring errors for Low Latency processes.