I believe you need to modify persistence.xml file for jBPM, as it defaults right now to JBoss AS 7. The file can be found in INSTALLER_HOME/db and change:
<
uncomment line for AS5 and comment out for AS7. Then rerun install demo and so on.
Alternatively you can find persistence.xml inside jbpm-gwt-console-server.war (jbpm-gwt-console-server-war/WEB-INF/classes/META-INF/) and to the mentioned changes.
HTH