One workflow process multiple files at the same time

[xliu]

Hi there, I have a question about how many threads one workflow can run?

The background is that there are hundreds of files we need to process every day, in the same format, but if we process the file one by one, it will cost about more than 8 hours. So we've set a main workflow to check files, and send files to a sub-workflow to process. The main workflow can call the sub-workflow multiple times, but the sub-workflow still process files one by one. 

Is there any other solution to handle this situation, or is there somewhere I can update on my workflows?

Thanks

[_Manoj_Kumar_]

Hello @xliu 

The workflow won't spawn more than one thread at a time. Because it is a sequential execution of the activities.

How often are you processing the files? In ACS, you can't schedule a workflow execution below 10 minutes.

The correct solution would be to merge those files into a single file and process it.

[xliu]

Hi Manoj,

Thanks for your reply!

We need to process data files everyday. So it seems like we need to set multiple workflows to process multiple data files at the same time?

Thanks,

Xihao

[Sukrity_Wadhwa]

Hi @_Manoj_Kumar_,

Can you help @xliu further with their request here?

Thanks!

[xliu · Accepted solution]

Hi Sukrity,

Thanks for the follow-up.

My solution for now is to use a master workflow to monitor/trigger several worker workflows to process multiple files at the same time. And they work as expected.

Please let me know if you have a better solution.

Thanks,

XH

[Sukrity_Wadhwa]

Thanks for sharing that. I would encourage our SMEs to chime in on this thread if they will find a better solution.

Thanks!