If you start with bin/run.bat you try to modify the line:
set JAVA_OPTS=%JAVA_OPTS% -Xms128m -Xmx512m
and increase the -Xmx value.
Also it might help to increase -Xms if you need the memory constantly (it will increase the initial heap size and free the JVM to recalculate the heap)