I did notice transaction problem with the example (http://community.jboss.org/people/bpmn2user/blog/2011/03/25/jbpm5--persistence-hello-process-example) if I use both the persistence units (process and task) together.
I have not tried the example with JBoss, but I presume it is something to do with the UserTransaction configuration as you had mentioned. As the JBPM5 console works on JBoss server (also JBPM console works fine with Tomcat/Oracle http://community.jboss.org/people/bpmn2user/blog/2011/02/26/orcale-integration-with-jbpm5), I think it has to do the UserTrasaction/JTA configuration.