Expand my Community achievements bar.

SOLVED

One workflow process multiple files at the same time

Avatar

Level 3

Hi there, I have a question about how many threads one workflow can run?

 

The background is that there are hundreds of files we need to process every day, in the same format, but if we process the file one by one, it will cost about more than 8 hours. So we've set a main workflow to check files, and send files to a sub-workflow to process. The main workflow can call the sub-workflow multiple times, but the sub-workflow still process files one by one. 

 

Is there any other solution to handle this situation, or is there somewhere I can update on my workflows?

 

Thanks

 

Topics

Topics help categorize Community content and increase your ability to discover relevant content.

1 Accepted Solution

Avatar

Correct answer by
Level 3

Hi Sukrity,

 

Thanks for the follow-up.

 

My solution for now is to use a master workflow to monitor/trigger several worker workflows to process multiple files at the same time. And they work as expected.

 

Please let me know if you have a better solution.

 

Thanks,

XH

View solution in original post

5 Replies

Avatar

Community Advisor

Hello @xliu 

 

The workflow won't spawn more than one thread at a time. Because it is a sequential execution of the activities.

 

How often are you processing the files? In ACS, you can't schedule a workflow execution below 10 minutes.

 

The correct solution would be to merge those files into a single file and process it.


     Manoj
     Find me on LinkedIn

Avatar

Level 3

Hi Manoj,

 

Thanks for your reply!

 

We need to process data files everyday. So it seems like we need to set multiple workflows to process multiple data files at the same time?

 

Thanks,

Xihao

Avatar

Administrator

Hi @_Manoj_Kumar_,

Can you help @xliu further with their request here?

Thanks!



Sukrity Wadhwa

Avatar

Correct answer by
Level 3

Hi Sukrity,

 

Thanks for the follow-up.

 

My solution for now is to use a master workflow to monitor/trigger several worker workflows to process multiple files at the same time. And they work as expected.

 

Please let me know if you have a better solution.

 

Thanks,

XH

Avatar

Administrator

Thanks for sharing that. I would encourage our SMEs to chime in on this thread if they will find a better solution.

Thanks!



Sukrity Wadhwa