Each application developed from aem or any product is different in nature there is no generic config you can set that works for all. Arrive to optimal config by analyzing outputs of [1] & your application. Sometimes increasing the memory is seen as a workaround to memory leaks or badly written software. More often than not this makes things worse in the long run and not better.
For valid application have seen
- Couple of customers with 8 or 16 gigs of heap size
- G1 collector supposedly performs better than any other algorithms. However seen with g1 the permgen usage going out of control might be due to some libraries doing dynamic classloading. See this article from Oracle for details at [2]. G1 doesn't do generational GC on the permgen space and only collects permgen on full gcs. The article also mentions that when you use CMS you can use the -XX:+CMSClassUnloadingEnabled option to cause the CMS gc to collect PermGen space in the CMS concurrent cycle.
Finally I would suggest to profiling of your application & testing out application with various option on some of the servers To find best suited.
[1]
* a heap dump from one of the instances with the issue. See this article on how to take a binary heap dump: http://helpx.adobe.com/experience-manager/kb/AnalyzeMemoryProblems.html
* thread dumps taken using jstack command http://helpx.adobe.com/experience-manager/kb/TakeThreadDump.html
* profiling data using the /system/console/profiler tool (run it for about 2-5 minutes while the performance is bad). Please see this article http://helpx.adobe.com/experience-manager/kb/AnalyzeUsingBuiltInProfiler.html
[2]
https://blogs.oracle.com/poonam/entry/about_g1_garbage_collector_permanent
