[wildfly-dev] Pooling EJB Session Beans per default

[Jason Greene]

On Aug 6, 2014, at 12:21 PM, Andrig Miller <anmiller at redhat.com> wrote:

> 
> 
> ----- Original Message -----
>> From: "Jason Greene" <jason.greene at redhat.com>
>> To: "Andrig Miller" <anmiller at redhat.com>
>> Cc: "Bill Burke" <bburke at redhat.com>, wildfly-dev at lists.jboss.org
>> Sent: Wednesday, August 6, 2014 11:08:02 AM
>> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
>> 
>> 
>> On Aug 6, 2014, at 10:49 AM, Andrig Miller <anmiller at redhat.com>
>> wrote:
>> 
>>> 
>>> 
>>> ----- Original Message -----
>>>> From: "Bill Burke" <bburke at redhat.com>
>>>> To: wildfly-dev at lists.jboss.org
>>>> Sent: Wednesday, August 6, 2014 9:30:06 AM
>>>> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default
>>>> 
>>> 


>>> This conversation is a perfect example of misinformation that
>>> causes us performance and scalability problems within our code
>>> bases.
>> 
>> It’s just a surprising result. The pool saves a few allocations, but
>> it also has the cost of concurrency usage which can trigger
>> blocking, additional barriers, and busy looping on CAS. You also
>> still have object churn in the underlying pool data structures that
>> occurs per invocation since every invocation is a check-out and a
>> check-in (requires a new node object instance), and if the semaphore
>> blocks you have additional allocation for the entry in the wait
>> queue. You factor in the remaining allocation savings relative to
>> other allocations that are required for the invocation, and it
>> should be a very small percentage. For that very small percentage to
>> lead to several times a difference in performance to me hints at
>> other factors being involved.
>> 
> 
> All logically thought through.  At a 15% lower transaction rate than we are doing now, we saw 4 Gigabytes per second of object allocation.  We, with Sanne doing most of the work, managed to get that down to 3 Gigabytes per second (I would have loved to get it to 2).  Much of that was Hibernate allocations, and of course that was with pooling on.  We have not spent the time to pinpoint the exact differences, memory and other, between having pooling on vs. off.  Our priority has been continue to scale the workload and fix any problems we see as a result.  We have managed to increase the transaction rate another 15% in the last couple of months, but still have another 17+% to go on a single JVM before we start looking at two JVM's for the testing.  
> 
> Once we get to our goal, I would love to put this on our list of tasks, so we can get the specific facts, and instead of talking theory, we will no exactly what can and cannot be done, and whether no pooling could ever match pooled.

Fair enough, and I certainly didn’t mean to imply that such work should be your team, I was just speaking generally. In any case, what I really really would like for us to achieve is a default implementation that performs generally well on all usage patterns, with no tuning required. Since we know that initialization can be costly for some applications usage of SLSB, such an implementation will definitely require a form of pooling. 

I suspect that a thread local based design with the pooling tied to worker threads will give us this. Alternatively a shared pool which is auto-tuned to match might be worth looking into. 

If there is anyone lurking who wishes to contribute in this area speak up, and I’ll worth with you on it. As doge would say “Such Fun. Much Glory” :) 

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat