After removing @ApplicationException from all existing SessionEJBs, and put the @ApplicationException in BusinessException (a sub of Exception) and SystemException (a sub of RuntimeException), the EAR loaded successfully in 5.1.0. Hope this info will help others who are doing migrating, too.
One thing is worth mentioning: the 4.2 allowed the @ApplicationException on SessionEJB, without complaining. It was not sure whether 4.2 just ignored the annoation, or it allows this as a variation of the systax. Further, with @ApplicationException on Exception, 5.1.0 loads the EAR successfully, but is there any independent test to verify that it will behave as it's supposed to? Let me know if you know the resutls or plan to do some test. Thanks.