Scenaio with a large amount of records to process times out after 40 minutes.

[PoppyJennings]

I have a scenario that is meant to take in a CSV file (picked up from Sharepoint as a starting trigger.) Based on the data in this spreadsheet, it is meant to create or update records in Salesforce.

The CSVs can have upto 100,000 rows of data. I was expecting that it could take some time to process all this data eg 5 hours or so. However, the scenario always stops working and crashes after 40 minutes.

Each row in the spreadsheet takes approx 1-2 seconds to process. I have tried breaking down the csvs into smaller amounts but even at 1000 rows of data it is hit and miss whether it completes or times out.

What can I do to ensure the scenario completes all exceutions for a large data amount?

[Doug_Den_Hoed__AtAppStore]

Hi @PoppyJennings,

We observed similar performance limitations when we created our Excel Updater solution (now well over ten years ago; poof!).

In addition to API throttling (ie limiting the number of concurrent connections - to 10 by default as I recall - and limiting the throughput of certain “expensive” calls), although our solutions predate and are independent of Fusion, we also a) increased the allowed file-size of (Excel) uploads, and b) increased the allowed time out wait time on our (non Fusion) web servers. For certain situations, we also resorted to trapping API errors and “retrying” (up to 5 times) to make our solutions more resilient,

As a result, our Excel Updater solution has been used to process tens of thousands of rows in a single upload, although - as you’d expected - it can take a considerable amount of time, proportionally.

I invite you to consider trying out our Excel Updater solution as a benchmark / workaround / permanent alternative (noting that your mileage may vary).

Regards,

Doug

[ChrisStephens]

Fusion has a hard time out at 40 minutes for any given scenario, that's not a configurable option, unfortunately. When I have run into similar situations I have typically fallen back to writing custom code (not Fusion, but like Python or what have you) that can be run independantly.

[AutomaSean]

Instead of creating smaller CSVs, I'd probably try using the original CSV but doing rows in chunks of 500. You could create a data store to store a value that represents which row to start on, and increment that value by 500 every time the scenario runs. First time it runs, start on row 1. Second time, 501. Etc until completion. Similar to how a website will show 50 results on a page and allow you to go to page 2, 3, etc.