Are you able to launch JBoss from the .bat file in the JBoss/Bin directory?
Try that, You may need to adjust the size parameters for JAVA_OPTS.
I have a laptop with 2 Gig of memory, and ended up with these settings;
set JAVA_OPTS=%JAVA_OPTS% -XX:PermSize=256m -XX:MaxPermSize=256m -XX:NewSize=250m -XX:MaxNewSize=250m -XX:SurvivorRatio=6 -Xms512m -Xmx512m.
Once you get JBoss to run from the .Bat file, you will need to apply the settings for PermSize, etc. to the JBoss service by editing the settings in your registry.
HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\JBoss for Adobe LiveCycle ES\Parameters
Good Luck
Mark