Error While uploading an asset in DAM

[bnagesh]

Hi All,

Version: AEM 6.5 SP 19 On prem hosted on Amazon AWS ec2 server

JDK: 11.0.27

OS: OEL 8.10 (recently migrated from CentOS 7)

Issue: When an user is trying to upload images in DAM (generally jpg/png) it shows Upload failed. This happens randomly, after multiple retries it finally uploads the image. Image size also are not very big its in KB only.

Analysis: We had gone through the error.log and workflow specific log during that time period and see no errors or exceptions logged in AEM author instance. Also one keynote is, this issue is being noticed only after recent OS migration. CPU utilization also is around 37% and never got any notifications for high memory or CPU during that time period.

Can anyone help us on what could be the potential issue, or any pointers as to where we need to look for the Root cause?

@arunpatidar @VeenaVikraman 

[SantoshSai]

Hi @bnagesh,

Looks like, it's highly likely the root cause lies in the OS-level or network-level interaction with AEM.

• AEM writes uploads first to the /tmp directory (or JVM-defined java.io.tmpdir).

• A slow or restricted /tmp mount can cause upload hangs or intermittent failures.

Please verify:

df -h /tmp
ls -ld /tmp
mount | grep tmp

• Ensure /tmp has enough space and isn't mounted with noexec or nosuid.

• You could also try temporarily changing AEM’s temp dir:

  -Djava.io.tmpdir=/opt/aem/tmp

  and observe upload behavior.

Hope this helps, if not I have 2 questions for you in advance:

1. Are you using a reverse proxy (Apache, Nginx, or AWS Load Balancer)?
2. Upload from inside the EC2 instance via localhost:4502 - does it succeed consistently?

[AmitVishwakarma]

Hi @bnagesh ,

Root Cause:

AEM writes uploaded files to the JVM’s temporary directory, usually /tmp.

If /tmp is:
  - Slow
  - Mounted with restrictive flags (noexec, nosuid, nodev)
  - Running out of space/inodes
then uploads silently fail — especially when Java or Asset workflows can't execute I/O reliably during the write phase.

This became an issue after OS migration because OS defaults and mount behavior changed.

Try below solution:
1. Create a Dedicated, Fast Temp Directory for AEM

sudo mkdir -p /opt/aem/tmp
sudo chown -R aemuser:aemgroup /opt/aem/tmp
chmod 755 /opt/aem/tmp
​

(Replace aemuser:aemgroup with the actual user/group running AEM.)

2. Modify JVM Start Parameters to Use New Temp Directory

Edit your AEM startup script (start or aem-author.service) and add:

-Djava.io.tmpdir=/opt/aem/tmp

For example, your command line might look like:

java -Xmx4096m -XX:+UseG1GC -Djava.io.tmpdir=/opt/aem/tmp -jar aem-author-p4502.jar

Validate

Upload assets (especially via AEM Touch UI).

Test from local network and remote clients.

Monitor /opt/aem/tmp to confirm file activity during uploads:

watch -n 1 ls -lh /opt/aem/tmp

Extra Diagnostics (Optional but Helpful)

Run the following to inspect /tmp:

df -h /tmp
mount | grep tmp
ls -ld /tmp

You’re looking for:
  - Low disk space/inodes?
  - Flags like noexec, nosuid, nodev?
  - Mounted on slow network/disk volume?
If yes, that’s why /tmp caused issues.

Regards,
Amit

[bnagesh]

Thank you @AmitVishwakarma  and @SantoshSai  Will check these and post further monitoring will update the thread if the issue got resolved.

[vikkip1]

default tmp path is updated but still we are seeing the asset upload failed.

and there is no issue with disc space

[arunpatidar]

Hi @bnagesh 

Can you check if there are console error in the browser when upload failed?

[vikkip1]

In the console and network we see bad gateway - https://st-cms.test.com/etc.clientlibs/clientlibs/granite/jquery/granite.js

POST https://st-cms.test.com/content/dam/we-retail/it.createasset.html 502 (Bad Gateway)

These are some warnings which we see in error.log - 08.05.2025 10:50:30.166 *WARN* [sling-threadpool-c61d514c-72a2-4eb6-ac37-dabf04eca987-(apache-sling-job-thread-pool)-16-Granite Workflow External Process Polling Queue(com/adobe/granite/workflow/external/polling/event)] org.apache.jackrabbit.oak.plugins.index.search.spi.query.FulltextIndex$FulltextPathCursor Index-Traversed 10000 nodes with filter Filter(query=select [jcr:path], [jcr:score], * from [dam:Asset] as a where [jcr:content/dam:scene7SanitizedPath] = '/content/dam/test-new/global/en_US/images/test/advanced-gas-path-7f (1).tif' and isdescendantnode(a, '/content/dam') /* xpath: /jcr:root/content/dam//element(*, dam:Asset)[(jcr:content/@dam:scene7SanitizedPath= '/content/dam/test-new/global/en_US/images/test/advanced-gas-path-7f(1).tif') ] */, path=/content/dam//*, property=[jcr:content/dam:scene7SanitizedPath=[/content/dam/test-new/global/en_US/images/test/advanced-gas-path-7f (1).tif]])
08.05.2025 10:50:30.652 *WARN* [sling-threadpool-c61d514c-72a2-4eb6-ac37-dabf04eca987-(apache-sling-job-thread-pool)-16-Granite Workflow External Process Polling Queue(com/adobe/granite/workflow/external/polling/event)] org.apache.jackrabbit.oak.plugins.index.search.spi.query.FulltextIndex$FulltextPathCursor Index-Traversed 20000 nodes with filter Filter(query=select [jcr:path], [jcr:score], * from [dam:Asset] as a where [jcr:content/dam:scene7SanitizedPath] = '/content/dam/test-new/global/en_US/images/test/advanced-gas-path-7f (1).tif' and isdescendantnode(a, '/content/dam') /* xpath: /jcr:root/content/dam//element(*, dam:Asset)[(jcr:content/@dam:scene7SanitizedPath= '/content/dam/test-new/global/en_US/images/test/advanced-gas-path-7f(1).tif') ] */, path=/content/dam//*, property=[jcr:content/dam:scene7SanitizedPath=[/content/dam/test-new/global/en_US/images/test/advanced-gas-path-7f (1).tif]])

[arunpatidar]

Hi @vikkip1 
Seems you are getting timeout.
You are on the right track, please check the error logs. Seems issue with index.

You can raise this with Adobe as well. 

[BharanNa]

Hi @arunpatidar Myself and @vikkip1 are working this with Adobe Support team as well, still there is no much progress. We are working on it, will mark the correct answer accodringly.

[aanchal-sikka]

@bnagesh Did you find the suggestions helpful? Please let us know if you require more information. Otherwise, please mark the answer as correct for posterity. If you've discovered a solution yourself, we would appreciate it if you could share it with the community. Thank you!