Sounds like a memory leak, I've had a similar problem where objects are not released and the heap become occupied.
In this case the GC will consume the time.
Do you can activate the GC logging?
See http://java.sun.com/docs/hotspot/gc5.0/gc_tuning_5.html for JVM options.
Also you can use 'jstat', part of java distribution with Java6 I'm not sure whether standard for Java5, to analyse the GC behaviour.