Hi,
'Out of memory' normaly not related to performance, it can caused by different reasons (see Javadoc can be thrown if any system resource is missed).
So please post the boot.log to see the complete JVM parameter. Also a gc.log might helpful (see http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html)
BTW what JVM version do you use?
There is no patch of JBoss to increase the performance, it is a complex issue ;)