I face the same problem at work and I use this tool:
http://www.eclipse.org/mat/
It gives you a good overview of the heap of your server.
You can also user the jconsole in the JDK to get an overview of the parts of the heap.
You also have to consider some issues, if you use individual portlets liferay:
1. If you use Spring and AnnotationConfigApplicationContext, then you must close the AnnotationConfigApplicationContext.
2. I make the experience, that SLSB are not destroyed. You must mark a methode with @PreDestroy and the call this method to remove the EJB.
3. If you use the EntityManager and Hibernate then clear it sometimes.
But that are only suggestions.