How to handle to large volume of assets in AEM

[girishBB]

Hello Everyone

We have a requirement to store 14,000+ assets ( XML, pdf) documents in AEM. Each file of very less size but the volume is huge.we dont use these assets to display on page/site.We don't publish the assets as well.

We are just uploading in AEM to handle metadata properties.
View asset content.

When uploaded on our lower environments. We saw multiple issues.

1. Timeline of assets was not working.
2. Aem automatically use to restart because of java out of memory exception
3. Aem was experiencing slowness

My question is, how to handle such large assets in AEM without impacting the performance of AEM?

Pls note:
we dont use these assets to display on page/site.
We don't publish the assets as well.

We are just uploading in AEM to handle metadata properties.
View asset content.

Any best approach or reference would help

@Jörg_Hoh 

Thanks

[aanchal-sikka · Accepted solution]

Hello @girishBB 

Please refer to https://experienceleague.adobe.com/docs/experience-manager-65/assets/administer/assets-migration-gui... . It provides the best practices for migrating assets.

Just to summarize:

1. Define a folder structure. Avoid adding more than 1000 assets in one folder.

2. If you are adding few assets at a time, you need not turn off workflows. But, if you are uploading in bulk (say >100 at a time), consider turning off DAM workflows. Once assets are upload, process them in batches via these workflows

The DAM workflows, specially "DAM update asset" workflow takes a lot of system resources. It creates renditions, extracts metadata from assets, skims though text for PDF for search.

Thus, when large amount of assets are ingested, a lot of workflows execute parallelly. This also impacts system performance.

Best is to use incremental approach for asset ingestion. For bulk ingestion, turn off DAM workflows, execute them in batches.

3. To have better performance, please refer https://experienceleague.adobe.com/docs/experience-manager-65/deploying/configuring/assets-performan... 

[FreedomMarketin]

I would note that 14k assets is not a large AEM Assets repository.  Most customers don't purchase AEM Assets until they have at least 10k assets.  I've seen the system scale to well over 10M assets and Adobe has indicated the system theoretically scales to a petabyte.  If you're having issues with only 14k assets, I would look at indexing.  The lack of proper indexing can sometimes be diagnosed by looking at Tools/Operations/Diagnosis/Request Performance (or Query Performance in the same menu path).  (I believe this option may only be available in AEM 6.5, but I don't think you should be seeing issues at this size repository with AEMaaCS.) 

The previous answer assumed that you were experiencing issues on the initial upload because you pushed too many assets in at once using an upload script that didn't rate limit the upload. If that's the case, the system should eventually catch up and become normally performant after a day or two.  If it's an ongoing issue after the upload, I would look at indexing.

Hope this helps,

Beau

[Jörg_Hoh]

What AEM version are you using? And are you storing these 14k assets within a single folder?