Best Practice for processing many bundles

Question

Hi All,this is rather generic question about how to best handle a high number of search results.The Workfront Search module in Fusion gives you the option to set a limit on the number of search results returned, but what if I need all matching records and the number can be high (several hundreds)? For example, I need to pull a list of all Assignments in a project and process each of them. For a large and complex project with many tasks this list can be extensive.How can I make sure that all matching records are returned and processed, while not risking to reach the 40 minutes runtime limit, etc.Is there a way to obtain and process them in batches?Any ideas are appreciated.Thanks,Tibor

Sven-iX · Accepted Answer

Hi @tibormolnar here is what I've used before:

First get a count of items, then get batches of 2000 records each until you exhaust all items
do the processing in a second scenario that is called from the first one, for each batch. This "multi-threads" your scenario.

Example:

customAPI module with the query but instead "search" action, use "count", gives you a number
setVar calculate the batches - you can go up to 2000 records or anything smaller
1. set batch size
2. cal number of batches = round( count / batchSize )
Repeater module with number of steps set to number of batches
setVar define start (the item to start the batch with) = the "i" from repeater * (batch size-1)
send these parameters to the "worker" scenario (ideally passing along the query from the count module)
in the worker, do the Search based on the query, start and batchSize
process the found items

Sign up

Login with SSO

Login to the community

Login with SSO

Scanning file for viruses.

This file cannot be downloaded