I was having a discussion with David and Lincoln yesterday about the https://jira.jboss.org/browse/JBAS-8557 issue which causes passivation failures. Although the issue with that specific JIRA appears to be a classloader problem, we ended up discussing some other things about passivation too. Basically, David suggests that we move away from JBSER library and instead use JBMAR for serializing. This may not completely get us past all the passivation issues (like this one https://jira.jboss.org/browse/JBAS-8384 and some other user reported issues) but from what I hear from David, JBMAR is certainly far better than JBSER.
I did a quick check in the ejb3-core code and it appears that the usage of JBSER during passivation is very limited. And from what I see, we should easily be able to move out these StatefulSessionPersistenceManager (that's the SPI) implementations out of ejb3-core, into separate projects. We could then easily have a JBMAR based implementation which we can plug in for AS6. For EAP5, we could continue using JBSER (if at all we want to do that).
P.S: The current priorities for EJB3 w.r.t AS6 probably won't allow this to happen for the upcoming CR. But I guess, it's better to start thinking about this and maybe pick this up after CR1 is released.