[jboss-as7-dev] Fwd: EJB passivation/activation, clustering and JPA extended persistence context handling...
Radoslaw Rodak
rodakr at gmx.ch
Mon Dec 5 18:02:51 EST 2011
Hi Scott
> I think its time to pull together some working examples of the edge cases (in the form of Arquillian unit tests). If anyone wants to contribute to the effort, http://community.jboss.org/wiki/HackingOnAS7 is good description of how to get starting with building AS7 (you get your own fork on github and everything, pretty cool ;). The tests could be similar to the other non-clustered tests in https://github.com/jbossas/jboss-as/tree/master/testsuite/integration/basic/src/test/java/org/jboss/as/test/integration/jpa and can be kept on a branch until we are ready to create clustered versions of the tests.
Thanks for helping how to start on jboss. I'm pretty fresh to Jboss AS project with all Sub Project modules, but I love how you guys are working.
I will be glad to help. Until end of year I'll be pretty busy, but later I will try to help to hack some stuff.
I'm hacking on App Servers since 1998 ( started with Inprise Borland Server ) and now my company decided to migrate next year from Oracle Weblogic 10.3.x to Jboss ....enforced with my sugestions to go for Java EE 6 so EAP 6.1 so for now AS 7.1CR1 snapshot ...
> I think your saying that the stateful session bean fail-over migration path from any one node, to another node, is pre-determined
Yes, but pre-determination doesn't need to be static. To which node the state should be replicated could be loadBalanced depends on policy.
Cluster Node Wight, Sever Request Queue/Backlog load usw..
> Such that if any node crashes, the stateful session beans on the crashed node, all migrate to one other predetermined node.
Depends how to understand "migrate on crash". For me, I would replicate the state of all SFSBs and all XPCs from one Node to another Node all the time. Mabe using hidden ejb View on other Node until crash is detected and on crash publish the view somwhere where client proxies will find it ( in weblogic you have something called cluster jndi tree wich is shared over all cluster nodes )
Fail Over needs to be fast and clear for clients. If clusters is under load with lot of clients say 100.. if you starts to distributes a lot of SFSBs and XPCs over all Cluster Member....first you spend time where to distribute, second you get 100 proxy requests where to finde all thos SFSB in Cluster... each of survived nodes get s 100 new Requests from 100 Clients from failed node if fail over loadBalanced over all nodes... instead just one node get hit by 100 clients. And then not realy hit, if shadow SFSBs and XPCs are ready to answer :-)
> The cluster would probably need enough nodes to keep the machines underutilized, in case they suddenly have to handle the load from one or more other nodes (wouldn't want to add load to an overloaded node that is already running 90% cpu utilization or almost out of Java memory already).
See my last comments...
In my experiance I never needed more the 4 Cluster Members. 75% of all cluster are 2 Node Clusters. In 9 big companies with havy load. :-)
When 4 Cluster Node were not good enought, there was allways a design problem in application :-)
Some Applikations where even slover using more cluster nodes ( you get more overhead... if you shared, synchronized ressources in cluster... )
> (wouldn't want to add load to an overloaded node that is already running 90% cpu utilization or almost out of Java memory already).
If all nodes in cluster have 90% cpu usage, the clustering works near at perfection. Then it doesn't mother to which nodes you will replicate and later fail over if needed. If one Node in cluster has 90% cpu Usage an others are IDLE douring StressTest then you need to reimpleted your LoadBalancing... because cluster is usless... loadBalacing works if Load is distributed over all Nodes. Nodes with more or less similar load.
> or almost out of Java memory already
I wonder how memory shortage is mesured in 7.1 :-) GarbageCollection is extrymly dynamic. Sometimes oa Node without Load can stay at high Memory usage until some Request requests more memory which might force full GC to free Memory.
Before JVM get's out of Memory, JVM respons will be extrymly slow ( Full GC -> not enought Memory - Full GC -> not Enough Memory ... )
If you have something which pings each cluster nodes and writes roundtripp time, then your Cluster LoadBalancing will redirect new Application Client Request to other nodes. Because "fail-over migration" is predeterineted, the cluster Member with "almost out of Memory" includes allready Memory for SFSB and XPCs state replication! So first you don't need additional Memory on, second if load balancing works, this node will not get new requests on it, if Cluster LoadBalancing works... and might recover.
If all Nodes are overloaded LoadBalancing of FailOver just make it wors... and then you better sugest to your Client to invest i additional RAM and/or more cluster Members :-)
Question is, what is chiepier and helps more ... buy memory or trouble shooting and playing with failover Load Balancing policy ...
> Every load balancing policy has positives and negatives. This one allows extended persistence context clustering, which is the positive.
Of cours. It allways good to have choice ( but less is more :-) )
Am 03.12.2011 um 05:14 schrieb Scott Marlow:
> Thanks for jumping in to help contribute solutions.
>
> I'm just looking for the edge cases that we have to support, with regard to clustering the extended persistence context. More cases are welcome, as that will help flesh out more issues.
>
> If I'm reading your suggestion correctly, I think your saying that the stateful session bean fail-over migration path from any one node, to another node, is pre-determined. Such that if any node crashes, the stateful session beans on the crashed node, all migrate to one other predetermined node. That would be more of a performance hit, than if we distributed the session beans evenly across the other remaining nodes but could work. The cluster would probably need enough nodes to keep the machines underutilized, in case they suddenly have to handle the load from one or more other nodes (wouldn't want to add load to an overloaded node that is already running 90% cpu utilization or almost out of Java memory already). Every load balancing policy has positives and negatives. This one allows extended persistence context clustering, which is the positive.
>
> I think its time to pull together some working examples of the edge cases (in the form of Arquillian unit tests). If anyone wants to contribute to the effort, http://community.jboss.org/wiki/HackingOnAS7 is good description of how to get starting with building AS7 (you get your own fork on github and everything, pretty cool ;). The tests could be similar to the other non-clustered tests in https://github.com/jbossas/jboss-as/tree/master/testsuite/integration/basic/src/test/java/org/jboss/as/test/integration/jpa and can be kept on a branch until we are ready to create clustered versions of the tests.
>
> On 12/02/2011 08:02 PM, Radoslaw Rodak wrote:
>> Sorry, should be:
>>
>> Let say 5 Node Cluster.
>>
>> Simpel Case without references between SFSBs
>> First Cluster Member with 4 independen SFSBs.
>>
>> SFSB1 uses XPC1
>> SFSB2 uses XPC2
>> SFSB3 uses XPC3
>> SFSB4 uses XPC4
>>
>> One remote Client connected to first Cluster Member keeping remote ref to all four SFSBs
>> First Node get killed ( kill -9 )
>>
>> If I understund Scott correctly this result in 4 LoadBalanced migration groups.
>> LoadBalancing "round robin" each SFSB goes to different cluster member.
>>
>> Remote Client needs then 4 new remote connections....does this need more time ????
>>
>> Assume now no LoadBalancing on failOver.
>>
>> All SFSBs fails over to same Cluster Member.
>>
>> Remote Client needs 1 new remote connection...
>> Don't need to figure out any relations ( faster ) because all SFSBs and all XPCs are on same Cluster Node.
>>
>> Cluster just defines which nodes replicates to which nodes. On FailOver One Cluster Member needs new Node for replication.
>>
>>
>> Anfang der weitergeleiteten E-Mail:
>>
>>> Von: Radoslaw Rodak<rodakr at gmx.ch>
>>> Datum: 3. Dezember 2011 01:49:28 MEZ
>>> An: jboss-as7-dev at lists.jboss.org
>>> Betreff: Re: [jboss-as7-dev] EJB passivation/activation, clustering and JPA extended persistence context handling...
>>>
>>> Let say 5 Node Cluster.
>>>
>>> Simpel Case without references between SFSBs
>>> First Cluster Member with 5 independen SFSBs.
>>>
>>> SFSB1 uses XPC1
>>> SFSB2 uses XPC2
>>> SFSB3 uses XPC3
>>> SFSB4 uses XPC4
>>>
>>> One remote Client connected to first Cluster Member keeping remote ref to all five SFSBs
>>> First Node killed ( kill -9 )
>>>
>>> If I understund Scott correctly this result in 4 LoadBalanced migration groups.
>>> Worst Case State replicated over 4 Nodes.
>>>
>>> Remote Client needs then 4 new remote connections....does this need more time ????
>>>
>>> Assume now no LoadBalancing on failOver.
>>>
>>> All SFSBs goest to let say second Cluster Member.
>>>
>>> Remote Client needs 1 new remote connection...
>>> Don't need to figure out any relations ( faster ) because all SFSBs and all XPCs are on same Cluster Node.
>>>
>>> Cluster just defines which nodes replicates to which nodes. On FailOver One Cluster Member needs new Node for replication.
>>>
>>>
>>> Am 02.12.2011 um 23:27 schrieb Brian Stansberry:
>>>
>>>> On 12/2/11 4:22 PM, Scott Marlow wrote:
>>>>> On 12/02/2011 04:38 PM, Jason T. Greene wrote:
>>>>>> On 12/2/11 2:55 PM, Scott Marlow wrote:
>>>>>>>> We also know that we need to replicate the serialization group (as its
>>>>>>>> definition changes dynamically). For example, the application has five
>>>>>>>> stateful session beans (SFSB1-SFSB5) that are associated with three
>>>>>>>> extended persistence contexts (XPC1-XPC3). The relationships between
>>>>>>>> these five beans are as follows:
>>>>>>>>
>>>>>>>> SFSB1 uses XPC1
>>>>>>>> SFSB2 uses XPC1 and has reference to SFSB3
>>>>>>>> SFSB3 uses XPC2
>>>>>>>> SFSB4 uses XPC2 and has reference to SFSB5
>>>>>>>> SFSB5 uses XPC3
>>>>>>>>
>>>>>>>> When SFSB1 fails over, we need to fail-over its working set (the
>>>>>>>> serialization group which includes SFSB1, SFSB2, SFSB3, SFSB4, SFSB5,
>>>>>>>> XPC1, XPC2, XPC3). In case its not obvious why, SFSB2 may have pending
>>>>>>>> database updates in XPC1, so it must fail-over also to the same target
>>>>>>>> node. Since SFSB2 has a reference to SFSB3, SFSB3 needs to fail-over
>>>>>>>> together with the group also (and so the pattern continues for SFSB4,
>>>>>>>> SFSB5, XPC3).
>>>>>>>
>>>>>>> The above is the application correctness issue. If SFSB2 fails over to
>>>>>>> a different node than SFSB4, the extended persistence context could be
>>>>>>> used on two different nodes. Basically, the extended persistence
>>>>>>> context can live through several non-transactional invocations. When a
>>>>>>> transaction is finally started on SFSB3 or SFSB4, the XPC2 changes will
>>>>>>> finally be committed.
>>>>>>
>>>>>> I still don't this is right. those references go across a proxy, and the
>>>>>> proxy's backing data can be anywhere.
>>>>>>
>>>>>
>>>>> I should of added that SFSB2 has a local reference to SFSB3 (which means
>>>>> they originates on the same JVM). I assumed that SFSB2 is migrated to
>>>>> the same target node as SFSB3 to keep the reference local (during a
>>>>> fail-over). No matter, where SFSB3 goes, SFSB4 needs to be on the same
>>>>> node (to share the same extended persistence context).
>>>>>
>>>>> The SFSBs themselves, can be anywhere but the extended persistence
>>>>> context, needs to stay with the SFSBs that inherit it.
>>>>>
>>>>> Maybe there is another way to tackle this that I'm not seeing but I
>>>>> wanted to mention these details to whomever is following this thread.
>>>>
>>>> BTW, is there any spec requirement that prohibits concurrent access to
>>>> an XPC?
>>>>
>>>> --
>>>> Brian Stansberry
>>>> Principal Software Engineer
>>>> JBoss by Red Hat
>>>> _______________________________________________
>>>> jboss-as7-dev mailing list
>>>> jboss-as7-dev at lists.jboss.org
>>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>>
>>>
>>> _______________________________________________
>>> jboss-as7-dev mailing list
>>> jboss-as7-dev at lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>>
>>
>>
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
More information about the jboss-as7-dev
mailing list