I had the same issue and solved it by upgrading to JDK 1.6+.
The user guide says JDK 1.5+ is all that is required and I happened to have 1.5.0_16 installed so I just went with that. After having the described issues (and noticing that in the demo video they are using JDK 1.6.0_14) I switched to JDK 1.6.0_16 since I had that installed already and now everything is working fine.
Also, I'm not sure if this was necessary or not but just to be safe, after switching JDKs I deleted the jbpm-installer/ folder, re-extracted the installer archive, and re-ran "ant install.demo".
Hope this helps.