It might be that objects survive too many GC's in youngGen and are moved to the OldGen, here the GC is more expensive, also each move in survivor spaces will have costs.
What might be an option is to use the CMS GC and increase the size of heap, specially the YoungGen that the objects are mostly out of date here.
Also it might that the survivor space is too small to keep all survived objects, in that case the whole bunch is directly moved to the OldGen.
I would check the behaviour in a test environment, you might use visualGC to see how objects are moved in survivor spaces, but be aware that visualGC will burden the watched VM.