Asset file split/chunk send over REST API - DAM

[Nitin_laad]

Hello Everybody, 

We have a requirement to push assets from AEM DAM to external DAM (non AEM-DAM) using rest API.
It is easy to push small size assets (up to 20 MB), but issue is with larger size assets due to Rest API data transfer limit.
One option is to split the large files in chunks and send each chunk over Rest api call.

but in case of bulk asset push what should be right approach to handle these assets chunks 

1. InputStream
2. File system (temp file)
3. or any other recommandation (please)

In case of inputStream, lets say for one assets (size = 50 mb), we have created 3 chunks, chunk push on Rest API should be sequential or parallel.
Sequential processing of chunks will take more time, whereas parallel processing will take less time, but  multi threading is having its own overhead. 

can you please suggest recommended way to handle this kind of use case.

Regards, 

Nj

[berliant]

You need to configure the chunkUploadMinFileSize property of the fileupload node in the CRX repository.

Please review:

https://docs.adobe.com/content/help/en/experience-manager-64/release-notes/sp-release-notes.html

and

https://helpx.adobe.com/ca/experience-manager/6-4/assets/using/managing-video-assets.html#Configurat...

[Nitin_laad]

Thank you Berliant for your time and reply.

I am not facing any asset size issue in uploading the asset in AEM DAM, but while pushing it to an external system using Rest API. 

Any reference material on that type of implementation would be a great help. 

Regards, 

Nj

[Jörg_Hoh · Accepted solution]

It totally depends on the remote system, what is the best way to make it work. Technically you can write code, which reads the data from the binary asset object, and sends it chunks of 20MB to the remote system. You just need to know if (and how) the remote system can handle that a new asset is sent in chunks.

Another way would be a transfer file share, where your custom code in AEM will export the files into and from where the remote system is reading the data then.

It's hard to recommend a specific approach without knowing the details of the affected products and all the circumstances and requirements around it.

[Nitin_laad]

Thank you Jorg for your time and reply. Remote system is able to merge asset chunks into one. I had posted this question in forum to get some reference material, if someone already implemented this type of use case.

[Jörg_Hoh]

I don't think that this is really specific to AEM. You can read any rendition as a stream from the repository, and from there on you should be able to use any sample code based on such an inputstream. I am not aware of a specific example code for AEM, sorry.