AEM publish servers Taking too long to stop

[AnkurAhlawat]

AEM publish servers are taking too long to stop, no error is getting printed in error log too. Some time it is taking upto 150 minutes + and then we need to kill the process. But have fear that it might corrupt the repository. Any help or pointers are appreciated.

[Opkar_Gill]

Hi,

it would help if you told us:

1. What version of AEM this is
2. Any configurations you have done i.e. external file datastore, "nocamplecontent"
3. Any hot fixes or service packs
4. Has this issue only started happening recently, have there been any changes to the repository: Code deployment, SP or hot fix

Regards,

Opkar

[AnkurAhlawat]

I am using AEM 6.1, with basic hotfixes. We are not using any external file store. We are using cold and standy architecture.

This issue has started from past 1 month only, and we have migrated from CQ5.6 to AEM 6.1 3 months back. For first two months we haven't faced this issue.

And this issue is not consistent on a specific server sometimes it happens on server 2 some time server 3. We were planning to put SP1 but we have put it on hold because of this.

[askdctm]

Hi Ankur,

Probably taking thread/heap dumps during that time frame will give some pointers.

[Opkar_Gill]

Hi,

You should raise a daycare ticket about this. Thread dumps will help to identify what is blocking shutdown in your instance. Did you do a code deployment just before this issue started?

Regards,

Opkar

[AnkurAhlawat]

We have already raised aday card ticket, and provided thread and heap dump. But didn't got any satisfactory response. That's why i have posted over here, is some one else is also facing same issue in 6.1 or not.

[cindyp42912329]

The same thing happened to our AEM publish instance too. We are running 6.2 with hot fixes.

Did you ever find out what triggered it and what is the solution?

[anilkum]

Hi,

Please check if there is an online compaction kicked off by default. If compaction is running then the shutdown will not happen. Also in case there is an reindexing triggered then shutdown will pause till it is completed. These situations I have seen but are not exhaustive,

Hope this helps.

Thanks,

Anil

[bobkranson]

Having seen your post the other day and looking through other things, I wonder if you have looked at how many threads are running?  I found this page in the default environment and noted the 'shutdown gracefully' setting which would be a reason why your system may take too long to shutdown (?!)  In the event there is a runaway thread, you would be waiting for default or internal timeouts to allow the 'graceful' shutdown.

http://localhost:4502/system/console/status-slingthreadpools

Most notable (and I would like some examples to set the 'Adobe-way') are these settings:

~- queue size : -1
- keep alive time : 60000
- block policy : RUN
- shutdown graceful : false
- shutdown wait time : -1

1) How to bound the queue size?  The thread pool might be 5 but you could have a million queued up!

2) Anyway to override the thread pools' shutdown graceful true/false during a stop request?  Me and grace had an argument, I want to kill her or end the relationship slamming the door shut no matter the cost....smile its referring to Process IDs (pids)

3) Anyway to inject a shutdown wait time of some value (2 or 5 mins) after a thread has started or for future threads?  I sent a letter to the IRS on Apr 15 but they haven't got it yet (shutdown notice), my returns will be late but sooner than they send me the late notice and bill....(wait time)

end of day, needed to inject smiles with these questions ;->

[bobkranson]

Having seen your post the other day and looking through other things, I wonder if you have looked at how many threads are running?  I found this page in the default environment and noted the 'shutdown gracefully' setting which would be a reason why your system may take too long to shutdown (?!)  In the event there is a runaway thread, you would be waiting for default or internal timeouts to allow the 'graceful' shutdown.

http://localhost:4502/system/console/status-slingthreadpools

Most notable (and I would like some examples to set the 'Adobe-way') are these settings:

~- queue size : -1
- keep alive time : 60000
- block policy : RUN
- shutdown graceful : false
- shutdown wait time : -1

1) How to bound the queue size?  The thread pool might be 5 but you could have a million queued up!

2) Anyway to override the thread pools' shutdown graceful true/false during a stop request?  Me and grace had an argument, I want to kill her or end the relationship slamming the door shut no matter the cost....smile its referring to Process IDs (pids)

3) Anyway to inject a shutdown wait time of some value (2 or 5 mins) after a thread has started or for future threads?  I sent a letter to the IRS on Apr 15 but they haven't got it yet (shutdown notice), my returns will be late but sooner than they send me the late notice and bill....(wait time)

end of day, needed to inject smiles with these questions ;->