Java Large Pages configuration and AEM Performance

[bobkranson]

The default java runtime allocates memory pages in 4k blocks.  There is a facility to enable 'large pages' whereby you can tune the size of the memory pages.  This supposedly relieves pressure on the CPU kernel to manipulate the memory pages (TLB) and *can* provide additional performance. (http://www.oracle.com/technetwork/java/javase/tech/largememory-jsp-137182.html)

As the TarMK stores everything in memory, I was looking for any AEM admin's out in the field that have tried and succeeded with configuring large pages (on Linux) and might share their experience(s) and results?  Is it working?  Does it add any performance?  Is there a point of diminishing return? (ie. vm size versus large page size versus repository size/configs that improve performance) (http://www.oracle.com/technetwork/articles/servers-storage-dev/hugepages-2099009.html)

[smacdonald2008 · Accepted solution]

I checked with Internal Adobe people - this was the reply:

TarMK uses memory mapped I/O so it stores chunks of the tar files in memory outside of the JVM.  The large pages option in the java runtime would have no impact on TarMK memory mapped I/O, however it might affect the large amount of segment ids stored in jvm heap by the TarMK.  It would also affect other aspects of AEM.  

[bobkranson]

Thank you for the inquiry and follow-up.  Any AEM users in the field who happen to flush out details on the impacts of large page configurations, please post us your info and results!

As a 'teaser' here is some info that I've developed which is 'unofficial' for my current client.  Your comments and feedback on this are welcome too!  This targets AEM 6.1 and AEM 6.2.

Memory Design

4gb
70% for app  (3g)
50% of the 70% for large pages (1.5g)

8gb (8589934590)
70% for app (6g, 6012954213)
50% of the 70% for large pages  (3g, 3006477106.5)

12gb (12884901885)
70% for app (9g, 9019431319.5)
50% of the 70% for large pages (4.5g, 4509715659.75)

Tune the Kernel Pages

First we will check the memory page size for the kernel.  This is usually always 4096 bytes (4k).  It factors into our large pages configuration.
    <linux host>:~ # getconf PAGE_SIZE
    4096

Next enable shared memory for that which is allocated to the application.

4 Gigabyte Machine
--------------------
3 GB = 3 * 1024 MB = 3072 MB = 3072 * 1024 kB = 3145728 kB and 3145728 kB * 1024b = 3221225472
# echo 3221225472 > /proc/sys/kernel/shmmax

3145728 kB / 2 = 1572864 kB / 2048 kB = 768:
# echo 768 > /proc/sys/vm/nr_hugepages

8 Gigabyte Machine
--------------------
# echo 6012954213 > /proc/sys/kernel/shmmax

6012954213 / 2 = 3006477106.5 / 1024 = 2936012 kb / 2048 kb = 1433
# echo 1433 > /proc/sys/vm/nr_hugepages

12 Gigabyte Machine
--------------------
# echo 9019431319 > /proc/sys/kernel/shmmax

9019431319 /2 = 4509715659.5 / 1024 = 4404019. kb / 2048 kb = 2150
# echo 2150 > /proc/sys/vm/nr_hugepages

Now we enable the Java 8 options:

These are already tested:
  CQ_JVM_OPTS='-server -Xms1024m -Xmx2560m -XX:+AggressiveOpts -XX:MaxMetaspaceSize=512m -XX:+UseAdaptiveGCBoundary -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:+ScavengeBeforeFullGC -XX:+ParallelRefProcEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:ParallelGCThreads=4 -XX:InitialCodeCacheSize=128M -XX:ReservedCodeCacheSize=128M -XX:NewRatio=2 -Djackrabbit.maxQueuedEvents=1000000 -Dcrx.memoryMaxUsage=98 -Dcrx.memoryMinStdev=1 -Dorg.apache.jackrabbit.core.state.validatehierarchy=true -Djava.awt.headless=true -XX:-UsePerfData'

These add the large pages:
  CQ_JVM_OPTS='-server -Xms1024m -Xmx2560m -XX:+AggressiveOpts -XX:+UseLargePages -XX:MaxMetaspaceSize=512m -XX:+UseAdaptiveGCBoundary -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 -XX:+CMSScavengeBeforeRemark -XX:+ScavengeBeforeFullGC -XX:+ParallelRefProcEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:ParallelGCThreads=4 -XX:InitialCodeCacheSize=128M -XX:ReservedCodeCacheSize=128M -XX:NewRatio=2 -Djackrabbit.maxQueuedEvents=1000000 -Dcrx.memoryMaxUsage=98 -Dcrx.memoryMinStdev=1 -Dorg.apache.jackrabbit.core.state.validatehierarchy=true -Djava.awt.headless=true -XX:-UsePerfData'

[Jörg_Hoh]

Hi,

I would not blindly change JVM settings just because it proved to be useful on some setup out there, see [1]. In most cases there is simply no benefit, in some cases it might help, while in the remaining ones it hurts performance and/or stability. If you plan to change a JVM parameter like this make sure that you have a performance test which you can use to prove that the change is benefical.

Jörg

[1] https://cqdump.wordpress.com/2013/07/29/jvm-tuning-or-premature-optimization/

[bobkranson]

Hi,

Yes.  I agree.  Do validate the change has value or numbers behind it.  In this case the idea is to proof whether creating large pages will improve performance of keeping tracking of many, many segment pointers in the JVM memory.