One workflow process multiple files at the same time | Community
Skip to main content
X
xliu
Level 3
December 6, 2023
Solved

One workflow process multiple files at the same time

  • December 6, 2023
  • 1 reply
  • 1703 views

Hi there, I have a question about how many threads one workflow can run?

 

The background is that there are hundreds of files we need to process every day, in the same format, but if we process the file one by one, it will cost about more than 8 hours. So we've set a main workflow to check files, and send files to a sub-workflow to process. The main workflow can call the sub-workflow multiple times, but the sub-workflow still process files one by one. 

 

Is there any other solution to handle this situation, or is there somewhere I can update on my workflows?

 

Thanks

 

This post is no longer active and is closed to new replies. Need help? Start a new post to ask your question.
Best answer by xliu

Hi @_manoj_kumar_,

Can you help @xliu further with their request here?

Thanks!


Hi Sukrity,

 

Thanks for the follow-up.

 

My solution for now is to use a master workflow to monitor/trigger several worker workflows to process multiple files at the same time. And they work as expected.

 

Please let me know if you have a better solution.

 

Thanks,

XH

1 reply

_Manoj_Kumar_
Community Advisor
_Manoj_Kumar_Community Advisor
Community Advisor
December 12, 2023

Hello @xliu 

 

The workflow won't spawn more than one thread at a time. Because it is a sequential execution of the activities.

 

How often are you processing the files? In ACS, you can't schedule a workflow execution below 10 minutes.

 

The correct solution would be to merge those files into a single file and process it.

     Manoj     Find me on LinkedIn
    X
    xliuAuthor
    Level 3
    December 14, 2023

    Hi Manoj,

     

    Thanks for your reply!

     

    We need to process data files everyday. So it seems like we need to set multiple workflows to process multiple data files at the same time?

     

    Thanks,

    Xihao

    Sukrity_Wadhwa
    Community Manager
    Sukrity_WadhwaCommunity Manager
    Community Manager
    January 10, 2024

    Hi @_manoj_kumar_,

    Can you help @xliu further with their request here?

    Thanks!

    Sukrity Wadhwa
    Terms & ConditionsAccessibility statement

    Loading footer...