How to run custom dam update workflow for 7 million images

[P_V_Nair]

We recently did a  oak sync of 7 milllion images from stage to prod and need to re-run custom dam update workflow to make sure all these images are loaded in prod dynamic media server and later activate all these images as well from author.

We were planning to batch them (which is really a tideous task on how to batch as we have deep folder inside our parent folder structure) and leverage ACS Commons Bulk workflow manager to do this in the next couple of weeks and later publish then after all these images are process. We are also not sure how long this will take and how prod instance will react.

Does anyone have any recommendations or any proven methods to bulk process these large set of images? Is their any easier or better way to do this activity in production?

Usually we used to run dam update workflow to the parent folder just after oak sync (but those were of less count like 20k or so) and then it used to complete in a night. But since the count here is 7 million, we are not sure if we can run this at parent folder level , as it can crash prod. Appreciate if anyone have any leads here.

[raushan123]

HI Nair,

we faced a similar situation but luckily we have only 20k images, we consulted with Adobe they recommended don't run the workflow on the parent folder, instead of this run the workflow on the subfolder to avoid the issue 

[P_V_Nair]

@raushan123  Our folders are deep to even level 7. So, this is going to be a very tedeous task.

[Sachin_Arora_]

@P_V_Nair Please confirm By oak sync you meant crx2oak sync?

I have done similar task for around 2 lakh assets which is way to less compared to 7 million in your case. As requirement is to sync Assets to DM and publish,I dont think you need to run DAM Update workflow as it do lot of other checks which is not needed like rendition creation etc.

You can create a custom workflow containing DM sync step or write a custom Job to sync assets to DM using APIs provided and publish asset based on its activation status. We used custom script approach earlier as there were some error in our case when we used DM sync processing step. 

You have to run this job in batches for sure as it will take good amount of time and also based on my experience, Scene7 processing queue is shared i.e. other clients might be using same service to sync asset to their DM server which also causes  delays based on load from different vendors.

[P_V_Nair]

@Sachin_Arora_ We are already in production and our dynamic media upload step is embedded in our custom dam update workflow. So we have to run dam update workflow itself to process them in dynamic media server. Also, we are running on a tight deadline , so don't have time to build something for this and deploy to production at this stage.

[Himanshu_Jain]

Scene7 queue is shared so make sure either you process in small batches as per your business priority or reach out to adobe support team to fine tune or allocate more resources for your program for scene7 sync.

Thanks

Himanshu