Best Practice for processing many bundles

[tibormolnar]

Hi All,

this is rather generic question about how to best handle a high number of search results.

The Workfront Search module in Fusion gives you the option to set a limit on the number of search results returned, but what if I need all matching records and the number can be high (several hundreds)? For example, I need to pull a list of all Assignments in a project and process each of them. For a large and complex project with many tasks this list can be extensive.

How can I make sure that all matching records are returned and processed, while not risking to reach the 40 minutes runtime limit, etc.

Is there a way to obtain and process them in batches?

Any ideas are appreciated.

Thanks,

Tibor

[Sven-iX · Accepted solution]

Hi @tibormolnar here is what I've used before: 

• First get a count of items, then get batches of 2000 records each until you exhaust all items
• do the processing in a second scenario that is called from the first one, for each batch. This "multi-threads" your scenario.

Example: 

1. customAPI module with the query but instead "search" action, use "count", gives you a number
2. setVar calculate the batches - you can go up to 2000 records or anything smaller
   1. set batch size
   2. cal number of batches = round( count / batchSize )
3. Repeater module with number of steps set to number of batches
4. setVar define start (the item to start the batch with) = the "i" from repeater * (batch size-1)
5. send these parameters to the "worker" scenario (ideally passing along the query from the count module)
6. in the worker, do the Search based on the query, start and batchSize
7. process the found items

[Doug_Den_Hoed__AtAppStore]

Hi @tibormolnar,

To spare you the day I recently lost when api-unsupported started returning needle duplicates among such haystack batches, I strongly urge you to ensure the request you use in step 3 (and optionally, step 1) from @Sven-iX is SORTED to avoid such duplicates and future proof your work.

Our www.atappstore.com lowest level plumbing now inspects every such API call to Workfront prior to execution and if the count exceeds the batch size but there is no "_Sort=" within, adds a magnetic "&ID_Sort=asc" to ensure Good Behavior.

Regards,

Doug 

[Sven-iX]

OMG - YES - sorting is a must, thank you for adding, @Doug_Den_Hoed__AtAppStore 
Had that experience too!
Seems weird the API doesn't already return a default sort... 

[tibormolnar]

Thanks for this Sven!

It all makes sense conceptually. I guess I just need to learn first how to call a scenario from within another scenario (or from outside of Fusion). If you happen to know where I best start reading about that, I appreciate the link. Otherwise I'll dig in the Community topics.

Thanks,

Tibor

[Sven-iX]

Oh that part is pretty simple:

in worker scenario, you start with a webhook. Copy the hook URL

In calling scenario you have a HTTP module and set the URL to the hook URL

pass what you need to pass as fields

[tibormolnar]

Ah, I see. I learnt something new today.

(I've only used Workfront event listener webhooks so far, now I just discovered the "Webhook" modules.)

Thanks!