Sorry, I can't help you with JRockit. But I do have some questions:
1) Why are you running 5.0.0.CR2? That is not even a released version. And even the released version of 5.0.0.GA was sort of flakey. You should really be running with at least 5.0.1.GA or better yet 5.1.0.GA.
2) Have you tried running with the Oracle JDK instead? If the problems are really with JRockit, then switching JVMs might help.
3) I have no idea about the ability of JRockit to perform a heap dump on an out of memory error. Can it do that? If so, have you done that and analyzed the dump to see which objects are taking up the heap space?
Randomly setting JVM options without knowing the root cause of the problem is fairly useless.