I'm not sure whether it is a JBoss4->JBoss5 problem, it is always difficult and restricted to use such (transactions, security, classloading).
I recommend to avoid such scenario!
You might check whether you have a correct classloading isolation of your app and provide the JBoss5 client libraries inside your application package.