I have the same issue. My JVM is allocated 5+ Gbs and jmap dies in much the same manner you describe.
My solution is to use gcore to produce a binary dump of your process, then use jmap to connect to that binary and convert it to an hprof dump that jhat can interpret.