It should be in the start up script that brings up AEM server with
CQ_JVM_OPTS='-server -Xmx1024m -XX:MaxPermSize=256M -Djava.awt.headless=true' in crx-quickstart/bin
Modify it to -Xmx2048m and test
If you work on local and start the server with cmd line/terminal then simply pass it in
java -Xmx2048M -jar <publish>.jar