It was caused by the QueryEngineSettings, .../system/console/jmx/org.apache.jackrabbit.oak%3Aname%3Dsettings%2Ctype%3DQuery… The limits can be set at runtime using JMX, domain org.apache.jackrabbit.oak type QueryEngineSettings, attribute names LimitInMemory and LimitReads.