The default persistence implementation uses JPA, where Hibernate can for example be used as the underlying persistence framework. So you can combine jBPM 5.0 persistence with an existing hibernate solution. If you really want to avoid using JPA altogether and use Hibernate directly, you could plug in your own persistence, where you replace the invocations to JPA in the implementation with direct invocation of the Hibernate equivalent.
Regarding transactions, jBPM 5.0 uses a pluggable mechanism here as well, where by default we use JTA but we also have a Spring implementation (as Spring doesn't provide access to the JTA transaction manager).
Kris