That error means that either you are running with too small of a heap or you have a memory leak. What heap size (-Xmx) are you specifying? And are you sure that the JVM is using that much heap? Checking the process memory utilization usually helps here - if you set -Xmx512M and the java process is using less than 100MB of memory, you know it didn't see the -Xmx512M. But if the java process is using around 600MB you kn ow that it did see the setting.
If you have a sufficiently large heap, then you need to figure out where the memory leak is. Try setting -XX:+HeapDumpOnOutOfMemoryError. This option causes the JVM to dump the heap to a file which you can then analyze using a tool such as VisualVM. VisualVM will show you which objects occur the most often and which objects are using the most heap space.
By the way, "too small of a heap" depends entirely on your application. If you are passing, and dealing with, very large objects then you will need a lot more heap that if you are working with ints, longs and short strings.