How to resolve out of memory error Java heap space

[Qamar_khan]

Hello All,

             We have implemented java sling servlet rest API for AEM 6.4 version. We are facing  out of memory error at regular time interval and server hanging.

Can someone help how to resolve it ? or is it due java sling servlet implementation or due to some server configuration issue.

Thanks

Qamar

[wasil_z]

Increase heap size first. Use correct memory parameters[1] either with jar or <crx-quickstart folder>/bin/start script when starting AEM instance.

Check your servlet/custom code for any possible memory leaks.

Thanks,

Wasil

[1] Increase java heap size – /home/edivad

[Qamar_khan]

Thanks Wasil for  quick reply,

1.Increase heap size first [I have done this]

After posting this I have found below solution and I have done this also just one question,Is warnings also reason of it?

2. Check your servlet/custom code for any possible memory leaks [I have done this]

I will try this and update you result

3.Use correct memory parameters[1] either with jar or <crx-quickstart folder>/bin/start script

Thanks

Qamar

[wasil_z]

Yes, warning in the logs - might be the reason. Analyse the logs carefully.

[Qamar_khan]

Hi Wasil,

             Eclipse compilation warnings logs or AEM Error.log? And can you help me which kind of parameter I need to set for

this solution

3.Use correct memory parameters[1] either with jar or <crx-quickstart folder>/bin/start script

Thanks

Qamar

[Qamar_khan]

Hi Wasil,

             I have checked the AEM Error logs carefully and observed the below error repeatedly occurring

03.01.2019 00:04:49.722 *ERROR* [sling-threadpool-18ec3542-a5b7-481d-acbf-03ea9a230a95-(apache-sling-job-thread-pool)-5-com_day_cq_replication_job_publish(com/day/cq/replication/job/publish)] com.day.cq.replication.Agent.publish Error while sending request: java.net.ConnectException: Connection refused: connect

03.01.2019 00:04:49.722 *ERROR* [sling-threadpool-18ec3542-a5b7-481d-acbf-03ea9a230a95-(apache-sling-job-thread-pool)-5-com_day_cq_replication_job_publish(com/day/cq/replication/job/publish)] com.day.cq.replication.Agent.publish Replication (DELETE) of /content/dam/asset/text/english/2-copy/ani/2C_EN not successful: java.net.ConnectException: Connection refused: connect Conversation follows

Is this mainly causing heap memory problem or server hanging?

Can please suggest solution for this com.day.cq.replication.Agent.publish Error.

Thanks

Qamar

[Opkar_Gill]

Hi Qamar,

make sure you have the minimum required amount of memory in your startup parameters, then use this page to Analyze Memory Problems.

Your error suggests the password in you replication may be incorrect. Can you open the agent and test the connection?

Regards,

Opkar

[Qamar_khan]

Thanks Opkar for reply,

            Earlier this post in our AEM6.4 we had increased the heap size in parameter of stat.bat file and quickstart.bat file as

if not defined CQ_JVM_OPTS set CQ_JVM_OPTS=-Xmx1024m -XX:MaxPermSize=1024M -Djava.awt.headless=true

which suggested in Analyze Memory Problems. ​.  But still we were facing the server hanging issue.

1.Secondly  we have tried this CQ_JVM_OPTS='-XX:+HeapDumpOnOutOfMemoryError' change which suggested in Analyze Memory Problems Page but in this case AEM were not stating.

2.For agent and test the connection from (/etc/replication/agents.author/publish.html)

we have tried but getting below attached error.

Can you please tell me what is going wrong in AEM6.4 server setting or java bundle.

Thanks

Qamar

[Opkar_Gill]

Hi Qamar,

the replication agent appears to be configured to point to localhost:4503, the default setting, which is used when running author and publish on the same local instance. If you are not running both on the same instance, add the correct url, also have you set the correct username and password for the publish instance? Follow this page to configure your Replication agent.

Regards,

Opkar

[Qamar_khan]

Hi Opkar,

             Currently we are using one single instance on localhost port 4502 which I am thinking is author instance.

We have checked publish instance 4503 url which is currently stop in our AEM.

With the help of this post https://helpx.adobe.com/experience-manager/kb/error-com-day-cq-replication-impl-AgentManagerImpl-Job... we have disable the agent.

Is there any more setting required ?

Thanks

Qamar

[Opkar_Gill]

Hi Qamar,

under the quickstart folder there is a foder "/crx-quickstart/threaddumps" which should contain thread dumps which you can analyse.

Also, how much memory does the server have and how much are you allocating to AEM.

What is your servlet doing?

Regards,

Opkar

[Qamar_khan]

Hi Opkar,

                 Currently with suggested changes by you and Wasil AEM operation looking normal with our customize java API bundle.

But we still checking memory with QA load testing to find out is it our problem resolved or not.

I have have allocated meomry AEM /crx-quickstart/quickstart and /start parameter  as

CQ_JVM_OPTS set CQ_JVM_OPTS=-Xmx1024m -XX:MaxPermSize=1024M.

Thanks both of you, I will update soon status of our problem with heap memory .

Thanks

Qamar

[Opkar_Gill]

Hi Qamar,

1GB is not much for a system, even a local instance. Have a read of section 3 (memory-mapped) in the following article CQ-OPS - What's New in AEM 6.0  to see how AEM uses the heap and available free memory to see why it's important to have enough memory allocated PLUS plenty of free memory as well.

Regards,

Opkar

[Qamar_khan]

Hi Opkar,

           Thanks for update, I try to increase size heap and as mentioned in section 3. And let you know our testing result.

Thanks

Qamar

[Qamar_khan]

Hi Opkar,

               We have observing very slow(almost dead) running behavior while uploading or retrieving the assets files operation from AEM CREX respiratory by our java bundle pro-grammatically. So can you please suggest

1.What could be ideal size heap memory for AEM6.4

2.Second what can causing this server slow running behavior in our case

Thanks

Qamar

[edubey]

Take a look at assets performance guide

https://helpx.adobe.com/experience-manager/6-4/sites/deploying/using/assets-performance-sizing.html

https://helpx.adobe.com/in/experience-manager/kb/AnalyzeMemoryProblems.html

[Qamar_khan]

Hi , Thanks for help will check and update how I can figure out solution by using first post.

As far second link is concerns we have taken almost all possible ways from it to resolved my problem.

Thanks

Qamar

[Opkar_Gill]

Hi Qamar,

edubey​ has already given the correct link to review. What you have to bear in mind is that Asset processing is very I/O and memory intensive, therefore you need to have the appropriate level of server resources in order to run such a system. Which was why I was asking about total memory and free memory.

In Assets heavy systems I have come across it is not unheard of to have 8-16GB of memory allocated to AEM with a total memory size of 32GB. If I were you I would start with 8GB and run tests from there. It is also very important to have enough free memory to be able to hold the segment store. So if you have 16GB free memory and your segment store is less than that size, it is good. Which is why you should have an external file datastore, so it does not increase the size of the segment store.

Regards,

Opkar

[raj_mandalapu]

Hi Qamar, Performance issue is not a simple thing and it depends on multiple factors.

1. What could be ideal size heap memory for AEM6.4

   It depends on mulitple factors.

1.      Total how many users are going to hit your server?
2.      How frequently you are updating content?
3. What is the DAM content size?

• etc.

2.Second what can causing this server slow running behavior in our case

When you upload Assets, internally the workflows will trigger, request you disable and try once. What is the size of Assets you are uploading? Please double check your code on closing resourceResolvers properly? is it happening on bulk upload or only for a single asset? How many users are parallelly accessing your server at a time?

You can download thread dumps and check where the problem is?

[Rahul17]

Hey you can increase your java heap size using this command just update your jar name.

java -Xmx1024m -jar aem-jar-name.jar