Which JVM are you using? OpenJDK? Or the one from Oracle? If from Oracle, revert to version 1.6.0_26 and see if that helps (we have noticed various issues with versions more recent than that). If that doesn't help, you'll have to contact Oracle to see if there is a memory leak issue with the JVM itself.
This almost sounds as if a system resource (such as a file handle or a socket connection) is not being released. But JBoss AS itself maintains pools of resources and reuses them, but who knows what the native JVM code is doing (only Oracle would know).