[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow commented on AS7-3229: ----------------------------------- I'll assign back to you after the XPC serialization testcase is runnable (after we switch to Hibernate 4.0.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow edited comment on AS7-3229 at 1/9/12 3:44 PM: ----------------------------------------------------------- I'll assign back to Paul after the XPC serialization testcase is runnable (after we switch to Hibernate 4.0.1 was (Author: smarlow): I'll assign back to you after the XPC serialization testcase is runnable (after we switch to Hibernate 4.0.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow commented on AS7-3229: ----------------------------------- The OptionalDataException went away after changing the test to only invoke the active node (instead of invoke the bean on node1+node2). I just saw a different exception (EJB Invocation failed on component StatefulBean for method public org.jboss.as.test.clustering.unmanaged.ejb3.xpc.bean.Employee org.jboss.as.test.clustering.unmanaged.ejb3.xpc.bean.StatefulBean.getEmployee(int): java.lang.IllegalArgumentException: object is not an instance of declaring class) with my current test changes. http://pastie.org/3161150 I'll soon close this jira if a fixable bug doesn't come out of this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow commented on AS7-3229: ----------------------------------- Will try adding a sleep after deploying to the other node/undeploying active node. This is part of the logic to simulate failover. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow updated AS7-3229: ------------------------------ Parent: AS7-2076 Issue Type: Sub-task (was: Bug) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow commented on AS7-3229: ----------------------------------- http://fpaste.org/JDwe/ shows why we deserialize twice (call stack from each of the calls). The second time comes from org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.postReplicate. The first comes from SerializationGroupMemberContainer.get the first serialization (after an invocation) comes from SerializationGroupMemberContainer.preReplicate (called from PassivatingBackingCacheImpl.release) and the second one is from SerializationGroupMemberContainer.update (also called from PassivatingBackingCacheImpl.release) http://fpaste.org/ZrEE/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow commented on AS7-3229: ----------------------------------- Weird, I got my first successful run ([StatefulWithXPCFailoverTestCase.java|https://github.com/scottmarlow/jboss...]) for no reason. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow commented on AS7-3229: ----------------------------------- Thinking more about it, since we are only putting stateful session beans in the SerializationGroup, its probably an instance StatefulBean, which looks like: {code:title=StatefulBean.java} @Clustered @javax.ejb.Stateful(name = "StatefulBean") public class StatefulBean implements Stateful { @PersistenceContext(unitName = "mypc", type = PersistenceContextType.EXTENDED) EntityManager em; @EJB SecondBean secondBean; /** * Create the employee but don't commit the change to the database, instead keep it in the * extended persistence context. * * @param name * @param address * @param id */ @Override @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) public void createEmployee(String name, String address, int id) { Employee emp = new Employee(); emp.setId(id); emp.setAddress(address); emp.setName(name); em.persist(emp); } @Override @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) public Employee getEmployee(int id) { return em.find(Employee.class, id, LockModeType.NONE); } @Override @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) public Employee getSecondBeanEmployee(int id) { return secondBean.getEmployee(id); } } {code} {code:title=SecondBean.java} @Clustered @javax.ejb.Stateful public class SecondBean { @PersistenceContext(unitName = "mypc", type = PersistenceContextType.EXTENDED) EntityManager em; @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) public Employee getEmployee(int id) { return em.find(Employee.class, id, LockModeType.NONE); } } {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow commented on AS7-3229: ----------------------------------- the first test run after making the 2 line change suggested by David, seemed to pass. https://github.com/scottmarlow/jboss-as/commit/aab8f780164d0d434911f5ff30... is the change. Will run more iterations in a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow commented on AS7-3229: ----------------------------------- Another observation is that org.jboss.as.ejb3.cache.impl.backing.SerializationGroupMemberContainer.postReplicate() is causing the deserialization to occur. This is interesting because its the second time the same buffer has been serialized. The first time we serialize the (same) buffer, is from SerializationGroupMemberContainer.get(). Paul and I discussed this the other day (http://pastie.org/3175394 contains the two call stacks). If deserializing the same buffer twice, fails on the second call (SerializationGroupMemberContainer.get()), that might mean that the serialized byte buffer has been corrupted in-between the two attempts to deserialize the same buffer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow commented on AS7-3229: ----------------------------------- PassivatingBackingCacheImpl.get(K key), is calling: E entry = store.get(key, lock); And then later: if (isClustered()) { passivationManager.postReplicate(entry); } The passivationManager.postReplicate(entry) method contains: // Restore the entry's ref to the group and object SerializationGroup<K, V, G> group = entry.getGroup(); if (group == null && entry.getGroupId() != null) { group = groupCache.get(entry.getGroupId()); } Perhaps we wouldn't be trying to deserialize the same buffer twice, if the first deserialization set the "group" in entry (if clustering is enabled). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow updated AS7-3229: ------------------------------ Attachment: firstcall.txt This is the first deserialized buffer (comes from the SerializationGroupMemberContainer.get call -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow commented on AS7-3229: ----------------------------------- The first set of attachments, are from when we got the "java.lang.IllegalArgumentException: object is not an instance of declaring class" I'll attach hex dumps for the OptionalDataException byte buffer too.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow updated AS7-3229: ------------------------------ Attachment: firstcall.txt secondcall.txt This is from a successful test run (happens occasionally). secondcall.txt is the hexdump of the deserialized serialization group (which is true for the other attachements). firstcall.txt is just the serialized bean. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow edited comment on AS7-3229 at 1/14/12 1:03 PM: ------------------------------------------------------------ Made progress on reducing the test down to a smaller test that still fails (playing on https://github.com/scottmarlow/jboss-as/tree/cluster9_hacktest). Will divide conquer some more to reduce further. After removing the XPC from the test, I still could reproduce the "java.lang.IllegalArgumentException: object is not an instance of declaring class" (that seemed to occur more often). By success, I mean that three consecutive test runs, passed. The following passes (the second clustered bean is not injected anymore which made the difference). I will add the XPC back in and if that works, will inject the second bean but try not tagging the second bean as clustered. {code} /* * JBoss, Home of Professional Open Source. * Copyright 2011, Red Hat, Inc., and individual contributors * as indicated by the @author tags. See the copyright.txt file in the * distribution for a full listing of individual contributors. * * This is free software; you can redistribute it and/or modify it * under the terms of the GNU Lesser General Public License as * published by the Free Software Foundation; either version 2.1 of * the License, or (at your option) any later version. * * This software is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. * * You should have received a copy of the GNU Lesser General Public * License along with this software; if not, write to the Free * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA * 02110-1301 USA, or see the FSF site: http://www.fsf.org. */ package org.jboss.as.test.clustering.unmanaged.ejb3.xpc.bean; import javax.ejb.TransactionAttribute; import javax.ejb.TransactionAttributeType; import org.jboss.ejb3.annotation.Clustered; /** * @author Paul Ferraro * @author Scott Marlow */ @Clustered @javax.ejb.Stateful(name = "StatefulBean") public class StatefulBean implements Stateful { // @PersistenceContext(unitName = "mypc", type = PersistenceContextType.EXTENDED) // EntityManager em; //@EJB SecondBean secondBean; private volatile Object share = null; /** * Create the employee but don't commit the change to the database, instead keep it in the * extended persistence context. * * @param name * @param address * @param id */ @Override @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) public void createEmployee(String name, String address, int id) { Employee emp = new Employee(); emp.setId(id); emp.setAddress(address); emp.setName(name); //em.persist(emp); share = emp; } @Override @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) public Employee getEmployee(int id) { //return em.find(Employee.class, id, LockModeType.NONE); return (Employee)share; } @Override @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) public Employee getSecondBeanEmployee(int id) { //secondBean.share(share); //return secondBean.getEmployee(id); return (Employee)share; } } {code} {code} /* * JBoss, Home of Professional Open Source. * Copyright 2012, Red Hat, Inc., and individual contributors * as indicated by the @author tags. See the copyright.txt file in the * distribution for a full listing of individual contributors. * * This is free software; you can redistribute it and/or modify it * under the terms of the GNU Lesser General Public License as * published by the Free Software Foundation; either version 2.1 of * the License, or (at your option) any later version. * * This software is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. * * You should have received a copy of the GNU Lesser General Public * License along with this software; if not, write to the Free * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA * 02110-1301 USA, or see the FSF site: http://www.fsf.org. */ package org.jboss.as.test.clustering.unmanaged.ejb3.xpc.bean; import javax.ejb.TransactionAttribute; import javax.ejb.TransactionAttributeType; import org.jboss.ejb3.annotation.Clustered; /** * test bean that uses the same extended persistence context as StatefulBean and therefor should always be able to * retrieve the same entities that are only in the extended persistence context (purposely not persisted to the database). * * @author Scott Marlow */ @Clustered @javax.ejb.Stateful public class SecondBean { // @PersistenceContext(unitName = "mypc", type = PersistenceContextType.EXTENDED) // EntityManager em; Object sharedObject = null; @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) public Employee getEmployee(int id) { //return em.find(Employee.class, id, LockModeType.NONE); return (Employee)sharedObject; } public void share(Object shared) { this.sharedObject = shared; } } {code} was (Author: smarlow): Made progress on reducing the test down to a smaller test that still fails (playing on https://github.com/scottmarlow/jboss-as/tree/cluster9_hacktest). Will divide conquer some more to reduce further. After removing the XPC from the test, I still could reproduce the "java.lang.IllegalArgumentException: object is not an instance of declaring class" (that seemed to occur more often). By success, I mean that three consecutive test runs, passed. The following passes (the second clustered bean is not injected anymore which made the difference). I will add the XPC back in and if that works, will inject the second bean but try not tagging the second bean as clustered. {code} /* * JBoss, Home of Professional Open Source. * Copyright 2011, Red Hat, Inc., and individual contributors * as indicated by the @author tags. See the copyright.txt file in the * distribution for a full listing of individual contributors. * * This is free software; you can redistribute it and/or modify it * under the terms of the GNU Lesser General Public License as * published by the Free Software Foundation; either version 2.1 of * the License, or (at your option) any later version. * * This software is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. * * You should have received a copy of the GNU Lesser General Public * License along with this software; if not, write to the Free * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA * 02110-1301 USA, or see the FSF site: http://www.fsf.org. */ package org.jboss.as.test.clustering.unmanaged.ejb3.xpc.bean; import javax.ejb.TransactionAttribute; import javax.ejb.TransactionAttributeType; import org.jboss.ejb3.annotation.Clustered; /** * @author Paul Ferraro * @author Scott Marlow */ @Clustered @javax.ejb.Stateful(name = "StatefulBean") public class StatefulBean implements Stateful { // @PersistenceContext(unitName = "mypc", type = PersistenceContextType.EXTENDED) // EntityManager em; //@EJB SecondBean secondBean; private volatile Object share = null; /** * Create the employee but don't commit the change to the database, instead keep it in the * extended persistence context. * * @param name * @param address * @param id */ @Override @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) public void createEmployee(String name, String address, int id) { Employee emp = new Employee(); emp.setId(id); emp.setAddress(address); emp.setName(name); //em.persist(emp); share = emp; } @Override @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) public Employee getEmployee(int id) { //return em.find(Employee.class, id, LockModeType.NONE); return (Employee)share; } @Override @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) public Employee getSecondBeanEmployee(int id) { //secondBean.share(share); //return secondBean.getEmployee(id); return (Employee)share; } } {code} /* * JBoss, Home of Professional Open Source. * Copyright 2012, Red Hat, Inc., and individual contributors * as indicated by the @author tags. See the copyright.txt file in the * distribution for a full listing of individual contributors. * * This is free software; you can redistribute it and/or modify it * under the terms of the GNU Lesser General Public License as * published by the Free Software Foundation; either version 2.1 of * the License, or (at your option) any later version. * * This software is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU * Lesser General Public License for more details. * * You should have received a copy of the GNU Lesser General Public * License along with this software; if not, write to the Free * Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA * 02110-1301 USA, or see the FSF site: http://www.fsf.org. */ package org.jboss.as.test.clustering.unmanaged.ejb3.xpc.bean; import javax.ejb.TransactionAttribute; import javax.ejb.TransactionAttributeType; import org.jboss.ejb3.annotation.Clustered; /** * test bean that uses the same extended persistence context as StatefulBean and therefor should always be able to * retrieve the same entities that are only in the extended persistence context (purposely not persisted to the database). * * @author Scott Marlow */ @Clustered @javax.ejb.Stateful public class SecondBean { // @PersistenceContext(unitName = "mypc", type = PersistenceContextType.EXTENDED) // EntityManager em; Object sharedObject = null; @TransactionAttribute(TransactionAttributeType.NOT_SUPPORTED) public Employee getEmployee(int id) { //return em.find(Employee.class, id, LockModeType.NONE); return (Employee)sharedObject; } public void share(Object shared) { this.sharedObject = shared; } } {code} {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow commented on AS7-3229: ----------------------------------- http://pastebin.com/mfKUsNEr is the latest IAE that I am getting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow commented on AS7-3229: ----------------------------------- This bug is only blocking my current test case, I could avoid nesting SFSBs in my test as a workaround. We should still fix the bug though. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow commented on AS7-3229: ----------------------------------- I changed my test case to avoid using a nested bean reference (just commented out for now, after this jira is fixed, can add it again). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Scott Marlow reassigned AS7-3229: --------------------------------- Assignee: Paul Ferraro (was: Scott Marlow) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Paul Ferraro commented on AS7-3229: ----------------------------------- Scott, are you still getting this error? SFSB serialization has changed a bit since this jira was opened - and I wonder if this is jira isn't obsolete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://issues.jboss.org/browse/AS7-3229?page=com.atlassian.jira.plugin.s... ] Stuart Douglas commented on AS7-3229: ------------------------------------- Can this be closed, or is there still something outstanding? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jboss.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira