Mixing JBoss version 4 <=> 5 is difficult, we give up because of transaction problems (JB5 calls JB4).
You might pack the jbossall-client.jar (for JB5 all needed client.jars) into the EAR deployed in JBoss4 with it's own classloader.
Also you need a EJB2 Home/Remote interface of your EJB3 because the error looks like EJB2 cast.