On Aug 6, 2014, at 12:21 PM, Andrig Miller <anmiller@redhat.com> wrote:

----- Original Message -----

From: "Jason Greene" <jason.greene@redhat.com>
To: "Andrig Miller" <anmiller@redhat.com>
Cc: "Bill Burke" <bburke@redhat.com>, wildfly-dev@lists.jboss.org
Sent: Wednesday, August 6, 2014 11:08:02 AM
Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default


On Aug 6, 2014, at 10:49 AM, Andrig Miller <anmiller@redhat.com>
wrote:

----- Original Message -----

From: "Bill Burke" <bburke@redhat.com>
To: wildfly-dev@lists.jboss.org
Sent: Wednesday, August 6, 2014 9:30:06 AM
Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default

This conversation is a perfect example of misinformation that
causes us performance and scalability problems within our code
bases.

It’s just a surprising result. The pool saves a few allocations, but
it also has the cost of concurrency usage which can trigger
blocking, additional barriers, and busy looping on CAS. You also
still have object churn in the underlying pool data structures that
occurs per invocation since every invocation is a check-out and a
check-in (requires a new node object instance), and if the semaphore
blocks you have additional allocation for the entry in the wait
queue. You factor in the remaining allocation savings relative to
other allocations that are required for the invocation, and it
should be a very small percentage. For that very small percentage to
lead to several times a difference in performance to me hints at
other factors being involved.

All logically thought through.  At a 15% lower transaction rate than we are doing now, we saw 4 Gigabytes per second of object allocation.  We, with Sanne doing most of the work, managed to get that down to 3 Gigabytes per second (I would have loved to get it to 2).  Much of that was Hibernate allocations, and of course that was with pooling on.  We have not spent the time to pinpoint the exact differences, memory and other, between having pooling on vs. off.  Our priority has been continue to scale the workload and fix any problems we see as a result.  We have managed to increase the transaction rate another 15% in the last couple of months, but still have another 17+% to go on a single JVM before we start looking at two JVM's for the testing.  

Once we get to our goal, I would love to put this on our list of tasks, so we can get the specific facts, and instead of talking theory, we will no exactly what can and cannot be done, and whether no pooling could ever match pooled.

Fair enough, and I certainly didn’t mean to imply that such work should be your team, I was just speaking generally. In any case, what I really really would like for us to achieve is a default implementation that performs generally well on all usage patterns, with no tuning required. Since we know that initialization can be costly for some applications usage of SLSB, such an implementation will definitely require a form of pooling.

I suspect that a thread local based design with the pooling tied to worker threads will give us this. Alternatively a shared pool which is auto-tuned to match might be worth looking into.

If there is anyone lurking who wishes to contribute in this area speak up, and I’ll worth with you on it. As doge would say “Such Fun. Much Glory” :)

--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat


_______________________________________________
wildfly-dev mailing list
wildfly-dev@lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev