I'd like to scale the send operation of my process dynamically. Looking for thoughts on how.
More details: i'm sending data to Netsuite using the netsuite connector. Netsuite has governors such that each license / integration user can only connect 10 times concurrently (before it begins rejecting the requests on a per user basis). Since this is based on the user and the user is configured on the connector, i have to configure multiple connectors (I don't want to do this); each hardwired to a specific integration user. This eats into licenses and makes for clutter in the process design, most importantly its hardwired at design time. This hardwiring of users to connectors and connector count to process results in an inelastic process. I would like the determination of how many concurrent connections and thus how many users to connect to be dynamic. for example
- Low load example: If i have 100 records to process, I'd like only one user to run 5 concurrent processes.
- High load example: in the same integration with no design time changes i'd like 10 users to run 10 concurrent connections each.
Furthermore i would like to share the integration users across multiple Netsuite integrations so i need some type of common / shared mechanism (like a connection pool). The issue here is i could be using integrationUser1 in three integrations that just happen to run at the same time. each uses 5 users, so now i have a problem (exceeded concurrent connections per Netsuite user governor). At the same time i could have 9 integration users doing nothing.
Thanks for any help or thoughts. Note: i expect i will have to do some customizations, but obviously the more out of box functionality i can use the better from a supportability perspective.