Throwing NotSerializableException instead of serializing means removing support for clustering SFSBs with an XPC. That IMHO would be a pretty major feature regression. I believe we have supported that since AS 4.2 if not earlier. On 10/24/11 9:39 AM, Scott Marlow wrote: -- Brian Stansberry Principal Software Engineer JBoss by Red Hat

On 10/24/11 9:54 AM, Scott Marlow wrote: Paul had done a bunch of cool stuff around XPC serialization that he had to put on the back burner to focus on the AS 6 Infinispan integration. -- Brian Stansberry Principal Software Engineer JBoss by Red Hat

I think its better to treat a clustered XPC as an 'individual SFSB'. Trying to maintain a graph around one XPC can quickly become very messy. Especially if different XPCs get involved. Conceptually the XPC can behave similar as a SFSB and we just keep proxies as references to it. Then it can replicate "at whim". Carlo On 10/24/2011 06:10 PM, Scott Marlow wrote:

Yeah, the graph could be deep. If we are replicating SFSB1 which is sharing an extended persistence context with SFSB2 and SFSB2 shares a separate extended persistence context with SFSB3 (and so it continues...). I don't think this is the likely 95% case but still possible. I'm open to alternative solutions. For fail-over to work, I think that all of the SFSBs in the graph, will also need to be available on the node that we fail over to. If we were to replicate them as a group, we could optimize the serialization of shared XPCs to only be sent once. I'm not saying that is the way we are going to do the SFSB level clustering, just pointing out that knowledge about the XPC use could be obtained through the proposed SPI (if we were to do it that way). I think we could handle replicating the SFSB graph entries individually and I can deal with that, if needed. Currently, we have the ExtendedEntityManager that is a proxy to the underlying entity manager. https://github.com/jbossas/jboss-as/blob/master/jpa/core/src/main/java/or... is the current implementation on AS7 master. https://github.com/scottmarlow/jboss-as/blob/2292b8393c1f32e23d949c728022... is an UUID based variation of the ExtendedEntityManager (on a cluster branch). For the passivation of SFSBs, I think the entire XPC inheritance derived graph could be passivated. If one of the SFSBs cannot be passivated (because its in a transaction, in use or perhaps has an extended persistence context entity that isn't serializeable), I wouldn't expect any of the SFSBs to be serialized. On 10/24/2011 02:57 PM, Carlo de Wolf wrote:

More about XPC clustering below. ;) On 11/11/2011 10:32 PM, Scott Marlow wrote: The above is the application correctness issue. If SFSB2 fails over to a different node than SFSB4, the extended persistence context could be used on two different nodes. Basically, the extended persistence context can live through several non-transactional invocations. When a transaction is finally started on SFSB3 or SFSB4, the XPC2 changes will finally be committed. We need to have a load balancing policy that knows how to fail-over the beans in the serialization group, as a group. So, that all requests to any of the beans, always run on the same target node (else we can have the same pending database changes applied from multiple nodes). We also need the serialization group to be complete. Scott

On 12/02/2011 04:38 PM, Jason T. Greene wrote: I should of added that SFSB2 has a local reference to SFSB3 (which means they originates on the same JVM). I assumed that SFSB2 is migrated to the same target node as SFSB3 to keep the reference local (during a fail-over). No matter, where SFSB3 goes, SFSB4 needs to be on the same node (to share the same extended persistence context). The SFSBs themselves, can be anywhere but the extended persistence context, needs to stay with the SFSBs that inherit it. Maybe there is another way to tackle this that I'm not seeing but I wanted to mention these details to whomever is following this thread.

On 12/02/2011 05:27 PM, Brian Stansberry wrote: Yes, section 7.2 seems to cover that (they are not expected to be thread-safe). JPA 2.0 specification section 7.2 Obtaining an EntityManager: " An entity manager must not be shared among multiple concurrently executing threads, as the entity manager and persistence context are not required to be threadsafe. Entity managers must only be accessed in a single-threaded manner. " The XPC usage model for sharing instances between SFSBs is described by section 7.6.2.1 Inheritance of Extended Persistence Context: " If a stateful session bean instantiates a stateful session bean (executing in the same EJB container instance) which also has such an extended persistence context, the extended persistence context of the first stateful session bean is inherited by the second stateful session bean and bound to it, and this rule recursively applies—independently of whether transactions are active or not at the point of the creation of the stateful session beans. If the persistence context has been inherited by any stateful session beans, the container does not close the persistence context until all such stateful session beans have been removed or otherwise destroyed. "

Sorry, should be: Let say 5 Node Cluster. Simpel Case without references between SFSBs First Cluster Member with 4 independen SFSBs. SFSB1 uses XPC1 SFSB2 uses XPC2 SFSB3 uses XPC3 SFSB4 uses XPC4 One remote Client connected to first Cluster Member keeping remote ref to all four SFSBs First Node get killed ( kill -9 ) If I understund Scott correctly this result in 4 LoadBalanced migration groups. LoadBalancing "round robin" each SFSB goes to different cluster member. Remote Client needs then 4 new remote connections....does this need more time ???? Assume now no LoadBalancing on failOver. All SFSBs fails over to same Cluster Member. Remote Client needs 1 new remote connection... Don't need to figure out any relations ( faster ) because all SFSBs and all XPCs are on same Cluster Node. Cluster just defines which nodes replicates to which nodes. On FailOver One Cluster Member needs new Node for replication. Anfang der weitergeleiteten E-Mail:

Hi Scott Thanks for helping how to start on jboss. I'm pretty fresh to Jboss AS project with all Sub Project modules, but I love how you guys are working. I will be glad to help. Until end of year I'll be pretty busy, but later I will try to help to hack some stuff. I'm hacking on App Servers since 1998 ( started with Inprise Borland Server ) and now my company decided to migrate next year from Oracle Weblogic 10.3.x to Jboss ....enforced with my sugestions to go for Java EE 6 so EAP 6.1 so for now AS 7.1CR1 snapshot ... Yes, but pre-determination doesn't need to be static. To which node the state should be replicated could be loadBalanced depends on policy. Cluster Node Wight, Sever Request Queue/Backlog load usw.. Depends how to understand "migrate on crash". For me, I would replicate the state of all SFSBs and all XPCs from one Node to another Node all the time. Mabe using hidden ejb View on other Node until crash is detected and on crash publish the view somwhere where client proxies will find it ( in weblogic you have something called cluster jndi tree wich is shared over all cluster nodes ) Fail Over needs to be fast and clear for clients. If clusters is under load with lot of clients say 100.. if you starts to distributes a lot of SFSBs and XPCs over all Cluster Member....first you spend time where to distribute, second you get 100 proxy requests where to finde all thos SFSB in Cluster... each of survived nodes get s 100 new Requests from 100 Clients from failed node if fail over loadBalanced over all nodes... instead just one node get hit by 100 clients. And then not realy hit, if shadow SFSBs and XPCs are ready to answer :-) See my last comments... In my experiance I never needed more the 4 Cluster Members. 75% of all cluster are 2 Node Clusters. In 9 big companies with havy load. :-) When 4 Cluster Node were not good enought, there was allways a design problem in application :-) Some Applikations where even slover using more cluster nodes ( you get more overhead... if you shared, synchronized ressources in cluster... ) If all nodes in cluster have 90% cpu usage, the clustering works near at perfection. Then it doesn't mother to which nodes you will replicate and later fail over if needed. If one Node in cluster has 90% cpu Usage an others are IDLE douring StressTest then you need to reimpleted your LoadBalancing... because cluster is usless... loadBalacing works if Load is distributed over all Nodes. Nodes with more or less similar load. I wonder how memory shortage is mesured in 7.1 :-) GarbageCollection is extrymly dynamic. Sometimes oa Node without Load can stay at high Memory usage until some Request requests more memory which might force full GC to free Memory. Before JVM get's out of Memory, JVM respons will be extrymly slow ( Full GC -> not enought Memory - Full GC -> not Enough Memory ... ) If you have something which pings each cluster nodes and writes roundtripp time, then your Cluster LoadBalancing will redirect new Application Client Request to other nodes. Because "fail-over migration" is predeterineted, the cluster Member with "almost out of Memory" includes allready Memory for SFSB and XPCs state replication! So first you don't need additional Memory on, second if load balancing works, this node will not get new requests on it, if Cluster LoadBalancing works... and might recover. If all Nodes are overloaded LoadBalancing of FailOver just make it wors... and then you better sugest to your Client to invest i additional RAM and/or more cluster Members :-) Question is, what is chiepier and helps more ... buy memory or trouble shooting and playing with failover Load Balancing policy ... Of cours. It allways good to have choice ( but less is more :-) ) Am 03.12.2011 um 05:14 schrieb Scott Marlow: > Thanks for jumping in to help contribute solutions. > > I'm just looking for the edge cases that we have to support, with regard to clustering the extended persistence context. More cases are welcome, as that will help flesh out more issues. > > If I'm reading your suggestion correctly, I think your saying that the stateful session bean fail-over migration path from any one node, to another node, is pre-determined. Such that if any node crashes, the stateful session beans on the crashed node, all migrate to one other predetermined node. That would be more of a performance hit, than if we distributed the session beans evenly across the other remaining nodes but could work. The cluster would probably need enough nodes to keep the machines underutilized, in case they suddenly have to handle the load from one or more other nodes (wouldn't want to add load to an overloaded node that is already running 90% cpu utilization or almost out of Java memory already). Every load balancing policy has positives and negatives. This one allows extended persistence context clustering, which is the positive. > > > On 12/02/2011 08:02 PM, Radoslaw Rodak wrote: >> Sorry, should be: >> >> Let say 5 Node Cluster. >> >> Simpel Case without references between SFSBs >> First Cluster Member with 4 independen SFSBs. >> >> SFSB1 uses XPC1 >> SFSB2 uses XPC2 >> SFSB3 uses XPC3 >> SFSB4 uses XPC4 >> >> One remote Client connected to first Cluster Member keeping remote ref to all four SFSBs >> First Node get killed ( kill -9 ) >> >> If I understund Scott correctly this result in 4 LoadBalanced migration groups. >> LoadBalancing "round robin" each SFSB goes to different cluster member. >> >> Remote Client needs then 4 new remote connections....does this need more time ???? >> >> Assume now no LoadBalancing on failOver. >> >> All SFSBs fails over to same Cluster Member. >> >> Remote Client needs 1 new remote connection... >> Don't need to figure out any relations ( faster ) because all SFSBs and all XPCs are on same Cluster Node. >> >> Cluster just defines which nodes replicates to which nodes. On FailOver One Cluster Member needs new Node for replication. >> >> >> Anfang der weitergeleiteten E-Mail: >> >>> Von: Radoslaw Rodak<rodakr(a)gmx.ch> >>> Datum: 3. Dezember 2011 01:49:28 MEZ >>> An: jboss-as7-dev(a)lists.jboss.org >>> Betreff: Re: [jboss-as7-dev] EJB passivation/activation, clustering and JPA extended persistence context handling... >>> >>> Let say 5 Node Cluster. >>> >>> Simpel Case without references between SFSBs >>> First Cluster Member with 5 independen SFSBs. >>> >>> SFSB1 uses XPC1 >>> SFSB2 uses XPC2 >>> SFSB3 uses XPC3 >>> SFSB4 uses XPC4 >>> >>> One remote Client connected to first Cluster Member keeping remote ref to all five SFSBs >>> First Node killed ( kill -9 ) >>> >>> If I understund Scott correctly this result in 4 LoadBalanced migration groups. >>> Worst Case State replicated over 4 Nodes. >>> >>> Remote Client needs then 4 new remote connections....does this need more time ???? >>> >>> Assume now no LoadBalancing on failOver. >>> >>> All SFSBs goest to let say second Cluster Member. >>> >>> Remote Client needs 1 new remote connection... >>> Don't need to figure out any relations ( faster ) because all SFSBs and all XPCs are on same Cluster Node. >>> >>> Cluster just defines which nodes replicates to which nodes. On FailOver One Cluster Member needs new Node for replication. >>> >>> >>> Am 02.12.2011 um 23:27 schrieb Brian Stansberry: >>> >>>> On 12/2/11 4:22 PM, Scott Marlow wrote: >>>>> On 12/02/2011 04:38 PM, Jason T. Greene wrote: >>>>>> On 12/2/11 2:55 PM, Scott Marlow wrote: >>>>>>>> We also know that we need to replicate the serialization group (as its >>>>>>>> definition changes dynamically). For example, the application has five >>>>>>>> stateful session beans (SFSB1-SFSB5) that are associated with three >>>>>>>> extended persistence contexts (XPC1-XPC3). The relationships between >>>>>>>> these five beans are as follows: >>>>>>>> >>>>>>>> SFSB1 uses XPC1 >>>>>>>> SFSB2 uses XPC1 and has reference to SFSB3 >>>>>>>> SFSB3 uses XPC2 >>>>>>>> SFSB4 uses XPC2 and has reference to SFSB5 >>>>>>>> SFSB5 uses XPC3 >>>>>>>> >>>>>>>> When SFSB1 fails over, we need to fail-over its working set (the >>>>>>>> serialization group which includes SFSB1, SFSB2, SFSB3, SFSB4, SFSB5, >>>>>>>> XPC1, XPC2, XPC3). In case its not obvious why, SFSB2 may have pending >>>>>>>> database updates in XPC1, so it must fail-over also to the same target >>>>>>>> node. Since SFSB2 has a reference to SFSB3, SFSB3 needs to fail-over >>>>>>>> together with the group also (and so the pattern continues for SFSB4, >>>>>>>> SFSB5, XPC3). >>>>>>> >>>>>>> The above is the application correctness issue. If SFSB2 fails over to >>>>>>> a different node than SFSB4, the extended persistence context could be >>>>>>> used on two different nodes. Basically, the extended persistence >>>>>>> context can live through several non-transactional invocations. When a >>>>>>> transaction is finally started on SFSB3 or SFSB4, the XPC2 changes will >>>>>>> finally be committed. >>>>>> >>>>>> I still don't this is right. those references go across a proxy, and the >>>>>> proxy's backing data can be anywhere. >>>>>> >>>>> >>>>> I should of added that SFSB2 has a local reference to SFSB3 (which means >>>>> they originates on the same JVM). I assumed that SFSB2 is migrated to >>>>> the same target node as SFSB3 to keep the reference local (during a >>>>> fail-over). No matter, where SFSB3 goes, SFSB4 needs to be on the same >>>>> node (to share the same extended persistence context). >>>>> >>>>> The SFSBs themselves, can be anywhere but the extended persistence >>>>> context, needs to stay with the SFSBs that inherit it. >>>>> >>>>> Maybe there is another way to tackle this that I'm not seeing but I >>>>> wanted to mention these details to whomever is following this thread. >>>> >>>> BTW, is there any spec requirement that prohibits concurrent access to >>>> an XPC? >>>> >>>> -- >>>> Brian Stansberry >>>> Principal Software Engineer >>>> JBoss by Red Hat >>>> _______________________________________________ >>>> jboss-as7-dev mailing list >>>> jboss-as7-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >>> >>> >>> _______________________________________________ >>> jboss-as7-dev mailing list >>> jboss-as7-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >> >> >> >> >> _______________________________________________ >> jboss-as7-dev mailing list >> jboss-as7-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev >