Expand my Community achievements bar.

SOLVED

Best practice to handle hundreds of workflows

Avatar

Level 2

What would be the best approach to recommend if a client has 300+ workflows running? in terms of performance

  1. What could indicate if a client is running 300+ workflows on their marketing instance? poor implementation? lack of product knowledge?
  2. In terms of performance issues? whats the best approach to handle such load?

At most, the max number of automated scheduled workflows I've seen running on a client is less than 100.

1 Accepted Solution

Avatar

Correct answer by
Employee Advisor

Hi Johanna,

So the number of scheduled workflows doesn't matter so much as to how many are executing at the same time.  So as long as these scheduled workflows are staggered to execute at different times then they should be fine performance wise.  The one exception to this is workflows in a paused status or using keep interim results as the build up in temp table data can cause significant performance issues even when not actively executing.

Now if the workflow executions are overlapping then we could potentially run into performance issues depending on the types of queries being run or the number of executions occurring at once.

If workflows are making inserts/updates to tables for example, then those tables get locked during that process, so if you have multiple workflows attempting to access a locked table then those workflows will need to wait for the inserts/updates to complete.

Whether or not there is poor implementation or lack of product knowledge would really require a discussion with the client to understand what the workflows are for and to see if there is a way to consolidate existing workflows.

If they have numerous data import/export workflows for example, it may make more sense to use external signals or jump points instead of schedulers to trigger off the next workflow versus having them on schedulers, that way you won't have any overlap in execution.

Unfortunately to recommend more would really require further review of the client's environment and a good understanding of what these workflows are trying to achieve.

Regards,

Craig

View solution in original post

1 Reply

Avatar

Correct answer by
Employee Advisor

Hi Johanna,

So the number of scheduled workflows doesn't matter so much as to how many are executing at the same time.  So as long as these scheduled workflows are staggered to execute at different times then they should be fine performance wise.  The one exception to this is workflows in a paused status or using keep interim results as the build up in temp table data can cause significant performance issues even when not actively executing.

Now if the workflow executions are overlapping then we could potentially run into performance issues depending on the types of queries being run or the number of executions occurring at once.

If workflows are making inserts/updates to tables for example, then those tables get locked during that process, so if you have multiple workflows attempting to access a locked table then those workflows will need to wait for the inserts/updates to complete.

Whether or not there is poor implementation or lack of product knowledge would really require a discussion with the client to understand what the workflows are for and to see if there is a way to consolidate existing workflows.

If they have numerous data import/export workflows for example, it may make more sense to use external signals or jump points instead of schedulers to trigger off the next workflow versus having them on schedulers, that way you won't have any overlap in execution.

Unfortunately to recommend more would really require further review of the client's environment and a good understanding of what these workflows are trying to achieve.

Regards,

Craig