Hi,
it seems that you give the JVM's more memory than the system has.
Could you give more details how the system is, do you mean you have two boxes with 8GB each?
How much JBoss instances do you start.
Also check the memory consumption of the JVM, you might use 'jstat -gc <JVM pid>' the output show how the memory is occupied. See Oracle page for jstat details, it is part of the JVM.
Also you hace 1.5GB PermSize does your application is that big? Maybe jstat show that the PermSize is too much and you can reduce it.