Hi, I want to discuss the topic of Session Bean Pooling in WildFly. I know that there was a discussion in the past to disable pooling of EJB Session Beans per default in WildFly. I understand when you argue that pooling a session bean is not faster than creating the bean from scratch each time a method is called. From the perspective of a application server developer this is a clear and easy decision. But from the view of an application developer this breaks one of the main concepts of session beans - the pooling. As a application developer I suppose my bean is pooled and I can use one of the life-cycle annotations to control my bean. This is a basic concept for all kind of beans. First I thought it could be a compromise to pool only these beans which have a life-cycle annotation. But this isn't a solution. Knowing that my bean will be pooled allows me - as a component developer - to use this as a caching mechanism. For example time intensive routines can also cache results in a local variable to be used next time a method is called. This isn't a bad practice and can increase performance of my component depending on the pool settings. So my suggestion is to pool also stateless session ejbs in the future. I guess form the specification there is no duty to pool beans. So there is nothing wrong when not pooling beans. And again I don't want to criticize. But at the end not pooling will decrease the performance of WildFly. Not the container itself but the applications running in WildFly. It takes me a long time to figure out why my application was a little bit slower in WildFly than in GlassFish until I recognized the missing pooling. I can activate pooling and everything is cool. But I guess some other application developers will only see that there application is slower in WildFly than on other application servers. And this will effect their decision. That is the argument to activate the pool per default. best regards Ralph

From: "Wolf-Dieter Fink" <wfink(a)redhat.com&gt; To: "Ralph Soika" <ralph.soika(a)imixs.com&gt;, wildfly-dev(a)lists.jboss.org Sent: Monday, July 21, 2014 11:39:12 AM Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default

Hi Ralph,

as I said in your community thread ( https://community.jboss.org/thread/242999 ) it is not only the problem to add a pool for it. At the moment there is only the strict-max-pool which limit the total number of beans pooled - and also the total number of beans which can be used in parallel. So that means adding the pool is a performance issue, which can make the situation better or worse depend on the application. So you need to adjust it depend on your needs.

If there is another implementation of a pool, i.e. which allow to have x pooled instances and more if needed, the pool might be back as default. For the moment it is makes no difference or is faster (in most cases) without pooling.

- Wolf

On 21/07/14 18:16, Ralph Soika wrote:

> Hi,

> I want to discuss the topic of Session Bean Pooling in WildFly. I > know that there was a discussion in the past to disable pooling of > EJB Session Beans per default in WildFly. > I understand when you argue that pooling a session bean is not > faster > than creating the bean from scratch each time a method is called. > From the perspective of a application server developer this is a > clear and easy decision. But from the view of an application > developer this breaks one of the main concepts of session beans - > the pooling. > As a application developer I suppose my bean is pooled and I can > use > one of the life-cycle annotations to control my bean. This is a > basic concept for all kind of beans. First I thought it could be a > compromise to pool only these beans which have a life-cycle > annotation. But this isn't a solution. > Knowing that my bean will be pooled allows me - as a component > developer - to use this as a caching mechanism. For example time > intensive routines can also cache results in a local variable to be > used next time a method is called. This isn't a bad practice and > can > increase performance of my component depending on the pool > settings.

> So my suggestion is to pool also stateless session ejbs in the > future. I guess form the specification there is no duty to pool > beans. So there is nothing wrong when not pooling beans. And again > I > don't want to criticize. But at the end not pooling will decrease > the performance of WildFly. Not the container itself but the > applications running in WildFly. > It takes me a long time to figure out why my application was a > little > bit slower in WildFly than in GlassFish until I recognized the > missing pooling. I can activate pooling and everything is cool. But > I guess some other application developers will only see that there > application is slower in WildFly than on other application servers. > And this will effect their decision. That is the argument to > activate > the pool per default.

> best regards > Ralph

_______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

Considering our benchmarking work through our performance team we have actually discovered that we actually need pooling of Stateless Session beans, because of memory allocation rates being too high. Allocating objects is fast in the latest JVM's, that's for sure, but you can easily create a situation where the memory allocation rate becomes too high, and you start getting excessive GC overhead. With that in mind, we have recently created (John O'Hara did the work from the middle-ware performance team), a new implementation of the StrictMaxPool. That implementation uses a ConcurrentLinkedQueue as the underlying data structure, as the original used a simple LinkedList with synchronization, so essentially all accesses to the pool were serialized. So, this didn't scale. The new implementation is upstream in Wildfly 9. There is no configuration change necessary, as it replaces the old implementation and is compatible. Like everything, there are tradeoffs, and the mantra that has spread recently that pooling is bad, is simply not the case. In some cases, allocating an object everytime is the best approach, in others cases pooling will be the best approach. Fortunately, you can do either in Wildfly (and EAP). Like always, you have to test your specific application, in the properly configured environment to know for sure. Andy ------------------------------------------------------------------------ *From: *"Wolf-Dieter Fink" <wfink(a)redhat.com&gt; *To: *"Ralph Soika" <ralph.soika(a)imixs.com&gt;, wildfly-dev(a)lists.jboss.org *Sent: *Monday, July 21, 2014 11:39:12 AM *Subject: *Re: [wildfly-dev] Pooling EJB Session Beans per default Hi Ralph, as I said in your community thread (https://community.jboss.org/thread/242999) it is not only the problem to add a pool for it. At the moment there is only the strict-max-pool which limit the total number of beans pooled - and also the total number of beans which can be used in parallel. So that means adding the pool is a performance issue, which can make the situation better or worse depend on the application. So you need to adjust it depend on your needs. If there is another implementation of a pool, i.e. which allow to have x pooled instances and more if needed, the pool might be back as default. For the moment it is makes no difference or is faster (in most cases) without pooling. - Wolf On 21/07/14 18:16, Ralph Soika wrote: Hi, I want to discuss the topic of Session Bean Pooling in WildFly. I know that there was a discussion in the past to disable pooling of EJB Session Beans per default in WildFly. I understand when you argue that pooling a session bean is not faster than creating the bean from scratch each time a method is called. From the perspective of a application server developer this is a clear and easy decision. But from the view of an application developer this breaks one of the main concepts of session beans - the pooling. As a application developer I suppose my bean is pooled and I can use one of the life-cycle annotations to control my bean. This is a basic concept for all kind of beans. First I thought it could be a compromise to pool only these beans which have a life-cycle annotation. But this isn't a solution. Knowing that my bean will be pooled allows me - as a component developer - to use this as a caching mechanism. For example time intensive routines can also cache results in a local variable to be used next time a method is called. This isn't a bad practice and can increase performance of my component depending on the pool settings. So my suggestion is to pool also stateless session ejbs in the future. I guess form the specification there is no duty to pool beans. So there is nothing wrong when not pooling beans. And again I don't want to criticize. But at the end not pooling will decrease the performance of WildFly. Not the container itself but the applications running in WildFly. It takes me a long time to figure out why my application was a little bit slower in WildFly than in GlassFish until I recognized the missing pooling. I can activate pooling and everything is cool. But I guess some other application developers will only see that there application is slower in WildFly than on other application servers. And this will effect their decision. That is the argument to activate the pool per default. best regards Ralph _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

On Mon, 2014-08-04 at 20:22 -0400, Bill Burke wrote: > I always liked the ThreadLocal pool. No synchronization, little to no > allocations. It also can cause massive memory leaks if invoked from threads which aren't re-used, like timer threads, and precautions aren't taken. AS/EAP 5 suffered from that problem with the default ThreadLocalPool. The major problem I've seen in support cases related to StrictMaxPool is the very arbitrary default size. I believe it used to be 30, and almost no-one knew that they might need to tune it to fit their applications' usage patterns and workload. Having a rising wait-time metric for the pool indicates that it may be too small, but I believe the statistics are disabled by default for performance reasons. InfinitePool obviously doesn't have that problem, but it would have been nicer if it had an idle period with expiry, like the JCA pools.

Your idea is something that I like as well. You have a min, but no max, and no blocking for an instance if there isn't one immediately available. The downside is its unbounded (I have also had customers tell me they needed a strict max option because they were delivering an application to their customers, and wanted to have strict control of the resources used), but it would be a nice compliment to what we have now.

On the old InfinitePool, using ThreadLocal, it was actually faster than the old StrictMaxPool, and it actually wasn't unbounded, because it would be bound by the thread pool size of the calling thread to the EJB instance. While it was faster than the old StrictMaxPool, it was probably because of the serialized nature of the old implementation. There is definitely room for additional implementations here. Andy ----- Original Message ----- > From: "Stuart Douglas" <stuart.w.douglas(a)gmail.com&gt; > To: "James Livingston" <jlivings(a)redhat.com&gt; > Cc: wildfly-dev(a)lists.jboss.org > Sent: Monday, August 4, 2014 7:51:15 PM > Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default > > I have often thought that a possible solution would be an unbounded > pool > that always keeps n instances around, but just creates new instances > instead of blocking if the pool is exhausted. In the majority of > cases > new instance creation will be way more performant than blocking. > > Stuart > > James Livingston wrote: >> On Mon, 2014-08-04 at 20:22 -0400, Bill Burke wrote: >>> I always liked the ThreadLocal pool. No synchronization, little >>> to no >>> allocations. >> >> It also can cause massive memory leaks if invoked from threads >> which >> aren't re-used, like timer threads, and precautions aren't taken. >> AS/EAP >> 5 suffered from that problem with the default ThreadLocalPool. >> >> The major problem I've seen in support cases related to >> StrictMaxPool is >> the very arbitrary default size. I believe it used to be 30, and >> almost >> no-one knew that they might need to tune it to fit their >> applications' >> usage patterns and workload. Having a rising wait-time metric for >> the >> pool indicates that it may be too small, but I believe the >> statistics >> are disabled by default for performance reasons. >> >> >> InfinitePool obviously doesn't have that problem, but it would have >> been >> nicer if it had an idle period with expiry, like the JCA pools. >> > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev > _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

From: "David M. Lloyd" <david.lloyd(a)redhat.com&gt; To: wildfly-dev(a)lists.jboss.org Sent: Tuesday, August 5, 2014 7:40:35 AM Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default I have always contended that the most appropriate place for throttling server usage is at the front end. If the number of incoming web and EJB requests can be limited, then everything else probably should just be unbounded (else you enter a configuration nightmare because *everything* has knobs). This way you can limit configuration to total incoming requests, and specific resources that have to be limited (like DB connections), and be assured of reasonable performance instead of having to read the encyclopedia to figure out all the tuning parameters you have to futz with. On 08/05/2014 08:07 AM, Andrig Miller wrote: > Your idea is something that I like as well. You have a min, but no > max, and no blocking for an instance if there isn't one > immediately available. The downside is its unbounded (I have also > had customers tell me they needed a strict max option because they > were delivering an application to their customers, and wanted to > have strict control of the resources used), but it would be a nice > compliment to what we have now. > > On the old InfinitePool, using ThreadLocal, it was actually faster > than the old StrictMaxPool, and it actually wasn't unbounded, > because it would be bound by the thread pool size of the calling > thread to the EJB instance. While it was faster than the old > StrictMaxPool, it was probably because of the serialized nature of > the old implementation. > > There is definitely room for additional implementations here. > > Andy > > ----- Original Message ----- >> From: "Stuart Douglas" <stuart.w.douglas(a)gmail.com&gt; >> To: "James Livingston" <jlivings(a)redhat.com&gt; >> Cc: wildfly-dev(a)lists.jboss.org >> Sent: Monday, August 4, 2014 7:51:15 PM >> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default >> >> I have often thought that a possible solution would be an >> unbounded >> pool >> that always keeps n instances around, but just creates new >> instances >> instead of blocking if the pool is exhausted. In the majority of >> cases >> new instance creation will be way more performant than blocking. >> >> Stuart >> >> James Livingston wrote: >>> On Mon, 2014-08-04 at 20:22 -0400, Bill Burke wrote: >>>> I always liked the ThreadLocal pool. No synchronization, >>>> little >>>> to no >>>> allocations. >>> >>> It also can cause massive memory leaks if invoked from threads >>> which >>> aren't re-used, like timer threads, and precautions aren't taken. >>> AS/EAP >>> 5 suffered from that problem with the default ThreadLocalPool. >>> >>> The major problem I've seen in support cases related to >>> StrictMaxPool is >>> the very arbitrary default size. I believe it used to be 30, and >>> almost >>> no-one knew that they might need to tune it to fit their >>> applications' >>> usage patterns and workload. Having a rising wait-time metric for >>> the >>> pool indicates that it may be too small, but I believe the >>> statistics >>> are disabled by default for performance reasons. >>> >>> >>> InfinitePool obviously doesn't have that problem, but it would >>> have >>> been >>> nicer if it had an idle period with expiry, like the JCA pools. >>> >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/wildfly-dev >> > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev > -- - DML _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

On Mon, 2014-08-04 at 20:22 -0400, Bill Burke wrote: > I always liked the ThreadLocal pool. No synchronization, little to no > allocations. It also can cause massive memory leaks if invoked from threads which aren't re-used, like timer threads, and precautions aren't taken. AS/EAP 5 suffered from that problem with the default ThreadLocalPool.

From: "Bill Burke" <bburke(a)redhat.com&gt; To: "James Livingston" <jlivings(a)redhat.com&gt; Cc: wildfly-dev(a)lists.jboss.org Sent: Tuesday, August 5, 2014 7:32:04 AM Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default On 8/4/2014 8:44 PM, James Livingston wrote: > On Mon, 2014-08-04 at 20:22 -0400, Bill Burke wrote: >> I always liked the ThreadLocal pool. No synchronization, little >> to no >> allocations. > > It also can cause massive memory leaks if invoked from threads > which > aren't re-used, like timer threads, and precautions aren't taken. > AS/EAP > 5 suffered from that problem with the default ThreadLocalPool. > Which is also something that could be mitigated by a common thread facility. Something AS/EAP 5 didn't have. Honestly though, I think this talk of EJB pooling is ridiculous. Component layers like CDI and JAX-RS don't have pooled architectures for their per-request objects. Also, in an average application, there are multiple orders of magnitude more non component classes that are being instantiated per request. Just think of all the Strings created by a simple HTTP request. You want a better focus? How about JSON/XML marshalling? Which could make things much easier to maintain then the hand-coded parsers that Wildfly uses to parse config. And much faster and less memory at runtime for SOAP and JAX-RS request that currently rely on java reflection. You could go research perfect hashing algorithms for URL matching with servlets and JAX-RS. You could go do some perf analysis of all of our frameworks to make memory reduction and speed recommendations or even Pull Requests. You could visit each major project and make sure our automated builds have and can run automated stress tests and are measured against previous versions. Or you could just focus on these silly benchmarks that test no-op HTTP and EJB requests.

-- Bill Burke JBoss, a division of Red Hat http://bill.burkecentral.com _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

> Those silly benchmarks, are indeed silly. Any workload that doesn't actually do anything with the requests is not very helpful. You can be really fast on them, and be really slow on a workload that actually does something. We have already seen this in one case recently. It’s all about precision, and limiting variables. I know its fun to criticize micro benchmarks (obviously micro is indeed relative because a great deal of processing goes on even when you have EJB and HTTP requests that don’t do much), however micro benchmarks, provided they are measured correctly, are quick to run and make it easy to tell what you have to fix. Large gigantic apps on the other hand are difficult to analyze, and do not necessarily represent activity that a differently architected app would show. To use an example, if you have pattern matching code, and you want it to be fast, it’s much easier (and accurate) to benchmark the pattern matching code then it is to benchmaark a full EE app that happens to use pattern matching in a few places. Arguing against targeted benchmarks is like arguing that you should never write a test case, because only real production load will show problems…. To some extent thats true, but that doesn’t mean that the problems you catch in simulations won’t affect production users.

On 8/5/2014 10:15 AM, Jason Greene wrote: >> Those silly benchmarks, are indeed silly. Any workload that doesn't actually do anything with the requests is not very helpful. You can be really fast on them, and be really slow on a workload that actually does something. We have already seen this in one case recently. > > It’s all about precision, and limiting variables. I know its fun to criticize micro benchmarks (obviously micro is indeed relative because a great deal of processing goes on even when you have EJB and HTTP requests that don’t do much), however micro benchmarks, provided they are measured correctly, are quick to run and make it easy to tell what you have to fix. Large gigantic apps on the other hand are difficult to analyze, and do not necessarily represent activity that a differently architected app would show. To use an example, if you have pattern matching code, and you want it to be fast, it’s much easier (and accurate) to benchmark the pattern matching code then it is to benchmaark a full EE app that happens to use pattern matching in a few places. > > Arguing against targeted benchmarks is like arguing that you should never write a test case, because only real production load will show problems…. To some extent thats true, but that doesn’t mean that the problems you catch in simulations won’t affect production users. > I'm more worried about focus rather than the actual benchmarks considering how resource strapped every project seems to be. I'd like to see a discussion started on what are the most important and *ripe* areas for optimization and then have the performance team validate and make recommendations on any priority list that this discussion generates.

On Aug 5, 2014, at 9:32 AM, Bill Burke <bburke(a)redhat.com&gt; wrote: > > > On 8/5/2014 10:15 AM, Jason Greene wrote: >>> Those silly benchmarks, are indeed silly. Any workload that doesn't actually do anything with the requests is not very helpful. You can be really fast on them, and be really slow on a workload that actually does something. We have already seen this in one case recently. >> >> It’s all about precision, and limiting variables. I know its fun to criticize micro benchmarks (obviously micro is indeed relative because a great deal of processing goes on even when you have EJB and HTTP requests that don’t do much), however micro benchmarks, provided they are measured correctly, are quick to run and make it easy to tell what you have to fix. Large gigantic apps on the other hand are difficult to analyze, and do not necessarily represent activity that a differently architected app would show. To use an example, if you have pattern matching code, and you want it to be fast, it’s much easier (and accurate) to benchmark the pattern matching code then it is to benchmaark a full EE app that happens to use pattern matching in a few places. >> >> Arguing against targeted benchmarks is like arguing that you should never write a test case, because only real production load will show problems…. To some extent thats true, but that doesn’t mean that the problems you catch in simulations won’t affect production users. >> > > I'm more worried about focus rather than the actual benchmarks considering how resource strapped every project seems to be. I'd like to see a discussion started on what are the most important and *ripe* areas for optimization and then have the performance team validate and make recommendations on any priority list that this discussion generates. The pooling concerns (aside from the occasional expensive post constructs) actually came from a large app/benchmark that the perf team was testing, and they were seeing an optimized strict max pool outperform no pooling at all, and it wasn’t post construct. Their theory is/was GC pressure, because this benchmark/app spends a lot of time in GC, and they see higher GC activity with no pooling. It’s possible it could be an indirect difference though, like the fact that strict max pool acts as a throttle might prevent the system from degrading as a result of no longer being able to keep up with the benchmark load.

Another possibility to look into is that I see we do: interceptorContext.setContextData(new HashMap<String, Object>()); *AND* private Map<Object, Object> instanceData = new HashMap<Object, Object>(); Aside from the unnecessary overhead, since JDK7, HashMap construction uses the murmur algorithm which requires using a random number generation under global locking. There were plans to optimize this, but it might not be in JDK7 yet. In a few places we use alternative map implementations to work around the issue.

> Its a horrible theory. :) How many EJB instances of a give type are > created per request? Generally only 1. 1 instance of one object of > one > type! My $5 bet is that if you went into EJB code and started > counting > how many object allocations were made per request, you'd lose count > very > quickly. Better yet, run a single remote EJB request through a perf > tool and let it count the number of allocations for you. It will be > greater than 1. :) > > Maybe the StrictMaxPool has an effect on performance because it > creates > a global synchronization bottleneck. Throughput is less and you end > up > having less concurrent per-request objects being allocated and GC'd. > The number per request, while relevant is only part of the story. The number of concurrent requests happening in the server dictates the object allocation rate. Given enough concurrency, even a very small number of object allocations per request can create an object allocation rate that can no longer be sustained.

From: "Radoslaw Rodak" <rodakr(a)gmx.ch&gt; To: wildfly-dev(a)lists.jboss.org Sent: Tuesday, August 5, 2014 6:51:03 PM Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default Am 06.08.2014 um 00:36 schrieb Bill Burke <bburke(a)redhat.com&gt;: > > > On 8/5/2014 3:54 PM, Andrig Miller wrote: >>> Its a horrible theory. :) How many EJB instances of a give type >>> are >>> created per request? Generally only 1. 1 instance of one object >>> of >>> one >>> type! My $5 bet is that if you went into EJB code and started >>> counting >>> how many object allocations were made per request, you'd lose >>> count >>> very >>> quickly. Better yet, run a single remote EJB request through a >>> perf >>> tool and let it count the number of allocations for you. It will >>> be >>> greater than 1. :) >>> >>> Maybe the StrictMaxPool has an effect on performance because it >>> creates >>> a global synchronization bottleneck. Throughput is less and you >>> end >>> up >>> having less concurrent per-request objects being allocated and >>> GC'd. >>> >> >> The number per request, while relevant is only part of the story. >> The number of concurrent requests happening in the server >> dictates the object allocation rate. Given enough concurrency, >> even a very small number of object allocations per request can >> create an object allocation rate that can no longer be sustained. >> > > I'm saying that the number of concurrent requests might not dictate > object allocation rate. There are probably a number of allocations > that > happen after the EJB instance is obtained. i.e. interception > chains, > contexts, etc. If StrictMaxPool blocks until a new instance is > available, then there would be less allocations per request as > blocking > threads would be serialized. > Scenarion 1 ) ------------------ Let say we have a pool of 100 Stateless EJBs and a constant Load of 50 Requests per second proceeded by 50 EJBs from the pool in one second. After 1000 seconds how many new EJB Instances will be created having a pool? answer 0 new EJBs worst case 100 EJB’s in pool… of course object allocation is much higher as of course 1 EJB call leads to many Object from one EJB but…let see situation without pool. 50 Request/s * 1000 seconds = worst case 50’ 000 EJB Instances on Java heap where 1 EJB might have many objects… as long as Garbage Collection was not triggered… which sounds to me like faster filling JVM heap and having ofter GC probable depending on GC Strategy. Scenarion 2) ------------------ Same as before, Load is still 50 Requests per second BUT EJB Method call takes 10s. after 10s we have 500 EJB Instances without pool, after 11s 550 - 10 = 540EJB Instances , after 12s 580 EJBs … after some time very bad perf…full GC …and mabe OutOfMemory.. So… performance advantage could also turn in to disadvantage :-) > Whoever is investigating StrictMaxPool, or EJB pooling in general > should > stop. Its pointless. Agree, pools are outdated…. but something like WorkManager for min, max Threads or even better always not less the X idle Threads would be useful :-) Radek

_______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

From: "Bill Burke" <bburke(a)redhat.com&gt; To: wildfly-dev(a)lists.jboss.org Sent: Wednesday, August 6, 2014 9:30:06 AM Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default On 8/6/2014 10:50 AM, Andrig Miller wrote: > > > ----- Original Message ----- >> From: "Radoslaw Rodak" <rodakr(a)gmx.ch&gt; >> To: wildfly-dev(a)lists.jboss.org >> Sent: Tuesday, August 5, 2014 6:51:03 PM >> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default >> >> >> Am 06.08.2014 um 00:36 schrieb Bill Burke <bburke(a)redhat.com&gt;: >> >>> >>> >>> On 8/5/2014 3:54 PM, Andrig Miller wrote: >>>>> Its a horrible theory. :) How many EJB instances of a give >>>>> type >>>>> are >>>>> created per request? Generally only 1. 1 instance of one >>>>> object >>>>> of >>>>> one >>>>> type! My $5 bet is that if you went into EJB code and started >>>>> counting >>>>> how many object allocations were made per request, you'd lose >>>>> count >>>>> very >>>>> quickly. Better yet, run a single remote EJB request through >>>>> a >>>>> perf >>>>> tool and let it count the number of allocations for you. It >>>>> will >>>>> be >>>>> greater than 1. :) >>>>> >>>>> Maybe the StrictMaxPool has an effect on performance because it >>>>> creates >>>>> a global synchronization bottleneck. Throughput is less and >>>>> you >>>>> end >>>>> up >>>>> having less concurrent per-request objects being allocated and >>>>> GC'd. >>>>> >>>> >>>> The number per request, while relevant is only part of the >>>> story. >>>> The number of concurrent requests happening in the server >>>> dictates the object allocation rate. Given enough concurrency, >>>> even a very small number of object allocations per request can >>>> create an object allocation rate that can no longer be >>>> sustained. >>>> >>> >>> I'm saying that the number of concurrent requests might not >>> dictate >>> object allocation rate. There are probably a number of >>> allocations >>> that >>> happen after the EJB instance is obtained. i.e. interception >>> chains, >>> contexts, etc. If StrictMaxPool blocks until a new instance is >>> available, then there would be less allocations per request as >>> blocking >>> threads would be serialized. >>> >> >> Scenarion 1 ) >> ------------------ >> Let say we have a pool of 100 Stateless EJBs and a constant Load >> of >> 50 Requests per second proceeded by 50 EJBs from the pool in >> one >> second. >> After 1000 seconds how many new EJB Instances will be created >> having >> a pool? answer 0 new EJBs worst case 100 EJB’s in pool… of course >> object allocation is much higher as of course 1 EJB call leads to >> many Object from one EJB but…let see situation without pool. >> >> 50 Request/s * 1000 seconds = worst case 50’ 000 EJB Instances on >> Java heap where 1 EJB might have many objects… as long as >> Garbage >> Collection was not triggered… which sounds to me like faster >> filling >> JVM heap and having ofter GC probable depending on GC Strategy. >> >> Scenarion 2) >> ------------------ >> Same as before, Load is still 50 Requests per second BUT EJB >> Method >> call takes 10s. >> after 10s we have 500 EJB Instances without pool, after 11s 550 - >> 10 >> = 540EJB Instances , after 12s 580 EJBs … after some time very >> bad >> perf…full GC …and mabe OutOfMemory.. >> >> So… performance advantage could also turn in to disadvantage :-) >> >> >>> Whoever is investigating StrictMaxPool, or EJB pooling in general >>> should >>> stop. Its pointless. >> >> Agree, pools are outdated…. but something like WorkManager for >> min, >> max Threads or even better always not less the X idle Threads >> would >> be useful :-) >> >> Radek >> > > The scenarios above are what is outddated. Fifty requests per > second isn't any load at all! We have 100's of thousands of > clients that we have to scale to, and lots more than 50 requests > per second. > What you mean to say is that you need to scale to 100's of thousands of clients on meaningless no-op benchmarks. :) I do know that that old SpecJ Java EE benchmarks artifically made EJB pooling important as process intensive calculation results were cached in these instances. But real-world apps don't use this feature/anti-pattern.

Also however crappy it was, I did implement an EJB container at one time in my career. :) I know for a fact that there are a number of per-request internal support objects that need to be allocated. Let's count: * The argument array (for reflection) * Each argument of the method call * The response object * Interceptor context object * The interceptor context attribute map * EJBContext * Subject, Principal, role mappings * Transaction context * The message object(s) specific to the remote EJB protocol Starts to add up huh? I'm probably missing a bunch more. Throw in interaction with JPA and you end up with even more per-request objects being allocated. You still believe pooling one EJB instance matters?

-- Bill Burke JBoss, a division of Red Hat http://bill.burkecentral.com _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

From: "Jason Greene" <jason.greene(a)redhat.com&gt; To: "Andrig Miller" <anmiller(a)redhat.com&gt; Cc: "Bill Burke" <bburke(a)redhat.com&gt;, wildfly-dev(a)lists.jboss.org Sent: Wednesday, August 6, 2014 11:08:02 AM Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default On Aug 6, 2014, at 10:49 AM, Andrig Miller <anmiller(a)redhat.com&gt; wrote: > > > ----- Original Message ----- >> From: "Bill Burke" <bburke(a)redhat.com&gt; >> To: wildfly-dev(a)lists.jboss.org >> Sent: Wednesday, August 6, 2014 9:30:06 AM >> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default >> >> >> >> On 8/6/2014 10:50 AM, Andrig Miller wrote: >>> >>> >>> ----- Original Message ----- >>>> From: "Radoslaw Rodak" <rodakr(a)gmx.ch&gt; >>>> To: wildfly-dev(a)lists.jboss.org >>>> Sent: Tuesday, August 5, 2014 6:51:03 PM >>>> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default >>>> >>>> >>>> Am 06.08.2014 um 00:36 schrieb Bill Burke <bburke(a)redhat.com&gt;: >>>> >>>>> >>>>> >>>>> On 8/5/2014 3:54 PM, Andrig Miller wrote: >>>>>>> Its a horrible theory. :) How many EJB instances of a give >>>>>>> type >>>>>>> are >>>>>>> created per request? Generally only 1. 1 instance of one >>>>>>> object >>>>>>> of >>>>>>> one >>>>>>> type! My $5 bet is that if you went into EJB code and >>>>>>> started >>>>>>> counting >>>>>>> how many object allocations were made per request, you'd lose >>>>>>> count >>>>>>> very >>>>>>> quickly. Better yet, run a single remote EJB request >>>>>>> through >>>>>>> a >>>>>>> perf >>>>>>> tool and let it count the number of allocations for you. It >>>>>>> will >>>>>>> be >>>>>>> greater than 1. :) >>>>>>> >>>>>>> Maybe the StrictMaxPool has an effect on performance because >>>>>>> it >>>>>>> creates >>>>>>> a global synchronization bottleneck. Throughput is less and >>>>>>> you >>>>>>> end >>>>>>> up >>>>>>> having less concurrent per-request objects being allocated >>>>>>> and >>>>>>> GC'd. >>>>>>> >>>>>> >>>>>> The number per request, while relevant is only part of the >>>>>> story. >>>>>> The number of concurrent requests happening in the server >>>>>> dictates the object allocation rate. Given enough >>>>>> concurrency, >>>>>> even a very small number of object allocations per request can >>>>>> create an object allocation rate that can no longer be >>>>>> sustained. >>>>>> >>>>> >>>>> I'm saying that the number of concurrent requests might not >>>>> dictate >>>>> object allocation rate. There are probably a number of >>>>> allocations >>>>> that >>>>> happen after the EJB instance is obtained. i.e. interception >>>>> chains, >>>>> contexts, etc. If StrictMaxPool blocks until a new instance >>>>> is >>>>> available, then there would be less allocations per request as >>>>> blocking >>>>> threads would be serialized. >>>>> >>>> >>>> Scenarion 1 ) >>>> ------------------ >>>> Let say we have a pool of 100 Stateless EJBs and a constant Load >>>> of >>>> 50 Requests per second proceeded by 50 EJBs from the pool in >>>> one >>>> second. >>>> After 1000 seconds how many new EJB Instances will be created >>>> having >>>> a pool? answer 0 new EJBs worst case 100 EJB’s in pool… of >>>> course >>>> object allocation is much higher as of course 1 EJB call leads >>>> to >>>> many Object from one EJB but…let see situation without pool. >>>> >>>> 50 Request/s * 1000 seconds = worst case 50’ 000 EJB Instances >>>> on >>>> Java heap where 1 EJB might have many objects… as long as >>>> Garbage >>>> Collection was not triggered… which sounds to me like faster >>>> filling >>>> JVM heap and having ofter GC probable depending on GC Strategy. >>>> >>>> Scenarion 2) >>>> ------------------ >>>> Same as before, Load is still 50 Requests per second BUT EJB >>>> Method >>>> call takes 10s. >>>> after 10s we have 500 EJB Instances without pool, after 11s 550 >>>> - >>>> 10 >>>> = 540EJB Instances , after 12s 580 EJBs … after some time very >>>> bad >>>> perf…full GC …and mabe OutOfMemory.. >>>> >>>> So… performance advantage could also turn in to disadvantage :-) >>>> >>>> >>>>> Whoever is investigating StrictMaxPool, or EJB pooling in >>>>> general >>>>> should >>>>> stop. Its pointless. >>>> >>>> Agree, pools are outdated…. but something like WorkManager for >>>> min, >>>> max Threads or even better always not less the X idle Threads >>>> would >>>> be useful :-) >>>> >>>> Radek >>>> >>> >>> The scenarios above are what is outddated. Fifty requests per >>> second isn't any load at all! We have 100's of thousands of >>> clients that we have to scale to, and lots more than 50 requests >>> per second. >>> >> What you mean to say is that you need to scale to 100's of >> thousands >> of >> clients on meaningless no-op benchmarks. :) I do know that that >> old >> SpecJ Java EE benchmarks artifically made EJB pooling important as >> process intensive calculation results were cached in these >> instances. >> But real-world apps don't use this feature/anti-pattern. >> > > I am not talking about a meaningless no-op benchmark, but a > benchmark that does lots of work. We don't use meaningless no-op > benchmarks on the performance team, with some exception for > microbenchmarks that we have carefully crafted that model the > interactions for a specific component within the context of how it > is actually used for a real application. > >> Also however crappy it was, I did implement an EJB container at >> one >> time >> in my career. :) I know for a fact that there are a number of >> per-request internal support objects that need to be allocated. >> Let's >> count: >> >> * The argument array (for reflection) >> * Each argument of the method call >> * The response object >> * Interceptor context object >> * The interceptor context attribute map >> * EJBContext >> * Subject, Principal, role mappings >> * Transaction context >> * The message object(s) specific to the remote EJB protocol >> >> Starts to add up huh? I'm probably missing a bunch more. Throw >> in >> interaction with JPA and you end up with even more per-request >> objects >> being allocated. You still believe pooling one EJB instance >> matters? >> > > See John O'Hara's post which shows our non-meaningless benchmark > and the difference that pooling makes vs. non-pooling. It is a > dramatic difference to say the least. There is certainly a correlation identified between the results of this benchmark and the use of pooling. However the underlying cause of the resulting difference is still unknown. If we knew definitively how and why this happens it would help in optimizing this further. As an example, if it turned out to be some secondary factor, like the throttling aspect of the pool, then eliminating these allocations (and others) with a zero-tuning approach, like thread local pooling would offer little to no improvement. If discovered it is indeed extreme object allocation, and that it came from thousands of nested calls in a request, then having a temporary per-request thread local cache would dramatically improve the results, and be cheap/quick to implement vs a full thread local solution. If there is a bug in our code somewhere where under certain situations we create hundreds of objects, when we should be creating 10s, and the pool covers that up, fixing that bug and removing the pool could lead to better results. If it turns out there is only 3% extra churn but that extra churn causes a 10x perf reduction in GC, then we better understand those limits and potentially work with the openjdk team in that area. > > This conversation is a perfect example of misinformation that > causes us performance and scalability problems within our code > bases. It’s just a surprising result. The pool saves a few allocations, but it also has the cost of concurrency usage which can trigger blocking, additional barriers, and busy looping on CAS. You also still have object churn in the underlying pool data structures that occurs per invocation since every invocation is a check-out and a check-in (requires a new node object instance), and if the semaphore blocks you have additional allocation for the entry in the wait queue. You factor in the remaining allocation savings relative to other allocations that are required for the invocation, and it should be a very small percentage. For that very small percentage to lead to several times a difference in performance to me hints at other factors being involved.

-- Jason T. Greene WildFly Lead / JBoss EAP Platform Architect JBoss, a division of Red Hat

On Aug 6, 2014, at 12:21 PM, Andrig Miller <anmiller(a)redhat.com&gt; wrote: > > > ----- Original Message ----- >> From: "Jason Greene" <jason.greene(a)redhat.com&gt; >> To: "Andrig Miller" <anmiller(a)redhat.com&gt; >> Cc: "Bill Burke" <bburke(a)redhat.com&gt;, wildfly-dev(a)lists.jboss.org >> Sent: Wednesday, August 6, 2014 11:08:02 AM >> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default >> >> >> On Aug 6, 2014, at 10:49 AM, Andrig Miller <anmiller(a)redhat.com&gt; >> wrote: >> >>> >>> >>> ----- Original Message ----- >>>> From: "Bill Burke" <bburke(a)redhat.com&gt; >>>> To: wildfly-dev(a)lists.jboss.org >>>> Sent: Wednesday, August 6, 2014 9:30:06 AM >>>> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default >>>> >>> >>> This conversation is a perfect example of misinformation that >>> causes us performance and scalability problems within our code >>> bases. >> >> It’s just a surprising result. The pool saves a few allocations, but >> it also has the cost of concurrency usage which can trigger >> blocking, additional barriers, and busy looping on CAS. You also >> still have object churn in the underlying pool data structures that >> occurs per invocation since every invocation is a check-out and a >> check-in (requires a new node object instance), and if the semaphore >> blocks you have additional allocation for the entry in the wait >> queue. You factor in the remaining allocation savings relative to >> other allocations that are required for the invocation, and it >> should be a very small percentage. For that very small percentage to >> lead to several times a difference in performance to me hints at >> other factors being involved. >> > > All logically thought through. At a 15% lower transaction rate than we are doing now, we saw 4 Gigabytes per second of object allocation. We, with Sanne doing most of the work, managed to get that down to 3 Gigabytes per second (I would have loved to get it to 2). Much of that was Hibernate allocations, and of course that was with pooling on. We have not spent the time to pinpoint the exact differences, memory and other, between having pooling on vs. off. Our priority has been continue to scale the workload and fix any problems we see as a result. We have managed to increase the transaction rate another 15% in the last couple of months, but still have another 17+% to go on a single JVM before we start looking at two JVM's for the testing. > > Once we get to our goal, I would love to put this on our list of tasks, so we can get the specific facts, and instead of talking theory, we will no exactly what can and cannot be done, and whether no pooling could ever match pooled. Fair enough, and I certainly didn’t mean to imply that such work should be your team, I was just speaking generally. In any case, what I really really would like for us to achieve is a default implementation that performs generally well on all usage patterns, with no tuning required. Since we know that initialization can be costly for some applications usage of SLSB, such an implementation will definitely require a form of pooling. I suspect that a thread local based design with the pooling tied to worker threads will give us this. Alternatively a shared pool which is auto-tuned to match might be worth looking into. If there is anyone lurking who wishes to contribute in this area speak up, and I’ll worth with you on it. As doge would say “Such Fun. Much Glory” :)

-- Jason T. Greene WildFly Lead / JBoss EAP Platform Architect JBoss, a division of Red Hat _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

----- Original Message ----- > From: "Jason Greene" <jason.greene(a)redhat.com&gt; > To: "Andrig Miller" <anmiller(a)redhat.com&gt; > Cc: "Bill Burke" <bburke(a)redhat.com&gt;, wildfly-dev(a)lists.jboss.org > Sent: Wednesday, August 6, 2014 11:08:02 AM > Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default > > > On Aug 6, 2014, at 10:49 AM, Andrig Miller <anmiller(a)redhat.com&gt; > wrote: > >> >> >> ----- Original Message ----- >>> From: "Bill Burke" <bburke(a)redhat.com&gt; >>> To: wildfly-dev(a)lists.jboss.org >>> Sent: Wednesday, August 6, 2014 9:30:06 AM >>> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default >>> >>> >>> >>> On 8/6/2014 10:50 AM, Andrig Miller wrote: >>>> >>>> >>>> ----- Original Message ----- >>>>> From: "Radoslaw Rodak" <rodakr(a)gmx.ch&gt; >>>>> To: wildfly-dev(a)lists.jboss.org >>>>> Sent: Tuesday, August 5, 2014 6:51:03 PM >>>>> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default >>>>> >>>>> >>>>> Am 06.08.2014 um 00:36 schrieb Bill Burke <bburke(a)redhat.com&gt;: >>>>> >>>>>> >>>>>> >>>>>> On 8/5/2014 3:54 PM, Andrig Miller wrote: >>>>>>>> Its a horrible theory. :) How many EJB instances of a give >>>>>>>> type >>>>>>>> are >>>>>>>> created per request? Generally only 1. 1 instance of one >>>>>>>> object >>>>>>>> of >>>>>>>> one >>>>>>>> type! My $5 bet is that if you went into EJB code and >>>>>>>> started >>>>>>>> counting >>>>>>>> how many object allocations were made per request, you'd lose >>>>>>>> count >>>>>>>> very >>>>>>>> quickly. Better yet, run a single remote EJB request >>>>>>>> through >>>>>>>> a >>>>>>>> perf >>>>>>>> tool and let it count the number of allocations for you. It >>>>>>>> will >>>>>>>> be >>>>>>>> greater than 1. :) >>>>>>>> >>>>>>>> Maybe the StrictMaxPool has an effect on performance because >>>>>>>> it >>>>>>>> creates >>>>>>>> a global synchronization bottleneck. Throughput is less and >>>>>>>> you >>>>>>>> end >>>>>>>> up >>>>>>>> having less concurrent per-request objects being allocated >>>>>>>> and >>>>>>>> GC'd. >>>>>>>> >>>>>>> >>>>>>> The number per request, while relevant is only part of the >>>>>>> story. >>>>>>> The number of concurrent requests happening in the server >>>>>>> dictates the object allocation rate. Given enough >>>>>>> concurrency, >>>>>>> even a very small number of object allocations per request can >>>>>>> create an object allocation rate that can no longer be >>>>>>> sustained. >>>>>>> >>>>>> >>>>>> I'm saying that the number of concurrent requests might not >>>>>> dictate >>>>>> object allocation rate. There are probably a number of >>>>>> allocations >>>>>> that >>>>>> happen after the EJB instance is obtained. i.e. interception >>>>>> chains, >>>>>> contexts, etc. If StrictMaxPool blocks until a new instance >>>>>> is >>>>>> available, then there would be less allocations per request as >>>>>> blocking >>>>>> threads would be serialized. >>>>>> >>>>> >>>>> Scenarion 1 ) >>>>> ------------------ >>>>> Let say we have a pool of 100 Stateless EJBs and a constant Load >>>>> of >>>>> 50 Requests per second proceeded by 50 EJBs from the pool in >>>>> one >>>>> second. >>>>> After 1000 seconds how many new EJB Instances will be created >>>>> having >>>>> a pool? answer 0 new EJBs worst case 100 EJB’s in pool… of >>>>> course >>>>> object allocation is much higher as of course 1 EJB call leads >>>>> to >>>>> many Object from one EJB but…let see situation without pool. >>>>> >>>>> 50 Request/s * 1000 seconds = worst case 50’ 000 EJB Instances >>>>> on >>>>> Java heap where 1 EJB might have many objects… as long as >>>>> Garbage >>>>> Collection was not triggered… which sounds to me like faster >>>>> filling >>>>> JVM heap and having ofter GC probable depending on GC Strategy. >>>>> >>>>> Scenarion 2) >>>>> ------------------ >>>>> Same as before, Load is still 50 Requests per second BUT EJB >>>>> Method >>>>> call takes 10s. >>>>> after 10s we have 500 EJB Instances without pool, after 11s 550 >>>>> - >>>>> 10 >>>>> = 540EJB Instances , after 12s 580 EJBs … after some time very >>>>> bad >>>>> perf…full GC …and mabe OutOfMemory.. >>>>> >>>>> So… performance advantage could also turn in to disadvantage :-) >>>>> >>>>> >>>>>> Whoever is investigating StrictMaxPool, or EJB pooling in >>>>>> general >>>>>> should >>>>>> stop. Its pointless. >>>>> >>>>> Agree, pools are outdated…. but something like WorkManager for >>>>> min, >>>>> max Threads or even better always not less the X idle Threads >>>>> would >>>>> be useful :-) >>>>> >>>>> Radek >>>>> >>>> >>>> The scenarios above are what is outddated. Fifty requests per >>>> second isn't any load at all! We have 100's of thousands of >>>> clients that we have to scale to, and lots more than 50 requests >>>> per second. >>>> >>> What you mean to say is that you need to scale to 100's of >>> thousands >>> of >>> clients on meaningless no-op benchmarks. :) I do know that that >>> old >>> SpecJ Java EE benchmarks artifically made EJB pooling important as >>> process intensive calculation results were cached in these >>> instances. >>> But real-world apps don't use this feature/anti-pattern. >>> >> >> I am not talking about a meaningless no-op benchmark, but a >> benchmark that does lots of work. We don't use meaningless no-op >> benchmarks on the performance team, with some exception for >> microbenchmarks that we have carefully crafted that model the >> interactions for a specific component within the context of how it >> is actually used for a real application. >> >>> Also however crappy it was, I did implement an EJB container at >>> one >>> time >>> in my career. :) I know for a fact that there are a number of >>> per-request internal support objects that need to be allocated. >>> Let's >>> count: >>> >>> * The argument array (for reflection) >>> * Each argument of the method call >>> * The response object >>> * Interceptor context object >>> * The interceptor context attribute map >>> * EJBContext >>> * Subject, Principal, role mappings >>> * Transaction context >>> * The message object(s) specific to the remote EJB protocol >>> >>> Starts to add up huh? I'm probably missing a bunch more. Throw >>> in >>> interaction with JPA and you end up with even more per-request >>> objects >>> being allocated. You still believe pooling one EJB instance >>> matters? >>> >> >> See John O'Hara's post which shows our non-meaningless benchmark >> and the difference that pooling makes vs. non-pooling. It is a >> dramatic difference to say the least. > > There is certainly a correlation identified between the results of > this benchmark and the use of pooling. However the underlying cause > of the resulting difference is still unknown. If we knew > definitively how and why this happens it would help in optimizing > this further. As an example, if it turned out to be some secondary > factor, like the throttling aspect of the pool, then eliminating > these allocations (and others) with a zero-tuning approach, like > thread local pooling would offer little to no improvement. If > discovered it is indeed extreme object allocation, and that it came > from thousands of nested calls in a request, then having a temporary > per-request thread local cache would dramatically improve the > results, and be cheap/quick to implement vs a full thread local > solution. If there is a bug in our code somewhere where under > certain situations we create hundreds of objects, when we should be > creating 10s, and the pool covers that up, fixing that bug and > removing the pool could lead to better results. If it turns out > there is only 3% extra churn but that extra churn causes a 10x perf > reduction in GC, then we better understand those limits and > potentially work with the openjdk team in that area. > > >> >> This conversation is a perfect example of misinformation that >> causes us performance and scalability problems within our code >> bases. > > It’s just a surprising result. The pool saves a few allocations, but > it also has the cost of concurrency usage which can trigger > blocking, additional barriers, and busy looping on CAS. You also > still have object churn in the underlying pool data structures that > occurs per invocation since every invocation is a check-out and a > check-in (requires a new node object instance), and if the semaphore > blocks you have additional allocation for the entry in the wait > queue. You factor in the remaining allocation savings relative to > other allocations that are required for the invocation, and it > should be a very small percentage. For that very small percentage to > lead to several times a difference in performance to me hints at > other factors being involved. > All logically thought through. At a 15% lower transaction rate than we are doing now, we saw 4 Gigabytes per second of object allocation. We, with Sanne doing most of the work, managed to get that down to 3 Gigabytes per second (I would have loved to get it to 2). Much of that was Hibernate allocations, and of course that was with pooling on. We have not spent the time to pinpoint the exact differences, memory and other, between having pooling on vs. off. Our priority has been continue to scale the workload and fix any problems we see as a result. We have managed to increase the transaction rate another 15% in the last couple of months, but still have another 17+% to go on a single JVM before we start looking at two JVM's for the testing. Once we get to our goal, I would love to put this on our list of tasks, so we can get the specific facts, and instead of talking theory, we will no exactly what can and cannot be done, and whether no pooling could ever match pooled.

Andy > -- > Jason T. Greene > WildFly Lead / JBoss EAP Platform Architect > JBoss, a division of Red Hat > > _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

From: "Jason Greene" <jason.greene(a)redhat.com&gt; To: "Bill Burke" <bburke(a)redhat.com&gt; Cc: wildfly-dev(a)lists.jboss.org Sent: Wednesday, August 6, 2014 11:37:07 AM Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default On Aug 6, 2014, at 10:30 AM, Bill Burke <bburke(a)redhat.com&gt; wrote: > What you mean to say is that you need to scale to 100's of > thousands of > clients on meaningless no-op benchmarks. :) I do know that that > old > SpecJ Java EE benchmarks artifically made EJB pooling important as > process intensive calculation results were cached in these > instances. > But real-world apps don't use this feature/anti-pattern. If the benchmark in question is doing that, that would most definitely explain this. I mean we know the following can perform poorly without pooling: 1. Expensive initialization in @PostConstruct

2. Lazy expensive initialization in a invocation (what you allude to above)

3. Expensive initialization in a constructor

Large allocations of objects count as expensive initialization. -- Jason T. Greene WildFly Lead / JBoss EAP Platform Architect JBoss, a division of Red Hat _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

From: "Bill Burke" <bburke(a)redhat.com&gt; To: "Andrig Miller" <anmiller(a)redhat.com&gt; Cc: wildfly-dev(a)lists.jboss.org, "Jason Greene" <jason.greene(a)redhat.com&gt; Sent: Tuesday, August 5, 2014 4:36:11 PM Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default On 8/5/2014 3:54 PM, Andrig Miller wrote: >> Its a horrible theory. :) How many EJB instances of a give type >> are >> created per request? Generally only 1. 1 instance of one object >> of >> one >> type! My $5 bet is that if you went into EJB code and started >> counting >> how many object allocations were made per request, you'd lose >> count >> very >> quickly. Better yet, run a single remote EJB request through a >> perf >> tool and let it count the number of allocations for you. It will >> be >> greater than 1. :) >> >> Maybe the StrictMaxPool has an effect on performance because it >> creates >> a global synchronization bottleneck. Throughput is less and you >> end >> up >> having less concurrent per-request objects being allocated and >> GC'd. >> > > The number per request, while relevant is only part of the story. > The number of concurrent requests happening in the server > dictates the object allocation rate. Given enough concurrency, > even a very small number of object allocations per request can > create an object allocation rate that can no longer be sustained. > I'm saying that the number of concurrent requests might not dictate object allocation rate. There are probably a number of allocations that happen after the EJB instance is obtained. i.e. interception chains, contexts, etc. If StrictMaxPool blocks until a new instance is available, then there would be less allocations per request as blocking threads would be serialized. Whoever is investigating StrictMaxPool, or EJB pooling in general should stop. Its pointless.

-- Bill Burke JBoss, a division of Red Hat http://bill.burkecentral.com

On 08/06/2014 03:47 PM, Andrig Miller wrote: > > ----- Original Message ----- > >> From: "Bill Burke" <bburke(a)redhat.com&gt; >> >> To: "Andrig Miller" >> <anmiller(a)redhat.com&gt; >> >> Cc: >> wildfly-dev(a)lists.jboss.org, "Jason Greene" <jason.greene(a)redhat.com&gt; >> >> Sent: Tuesday, August 5, 2014 4:36:11 PM >> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default >> >> >> >> On 8/5/2014 3:54 PM, Andrig Miller wrote: >> >>>> Its a horrible theory. :) How many EJB instances of a give type >>>> are >>>> created per request? Generally only 1. 1 instance of one object >>>> of >>>> one >>>> type! My $5 bet is that if you went into EJB code and started >>>> counting >>>> how many object allocations were made per request, you'd lose >>>> count >>>> very >>>> quickly. Better yet, run a single remote EJB request through a >>>> perf >>>> tool and let it count the number of allocations for you. It will >>>> be >>>> greater than 1. :) >>>> >>>> Maybe the StrictMaxPool has an effect on performance because it >>>> creates >>>> a global synchronization bottleneck. Throughput is less and you >>>> end >>>> up >>>> having less concurrent per-request objects being allocated and >>>> GC'd. >>>> >>>> >>> The number per request, while relevant is only part of the story. >>> The number of concurrent requests happening in the server >>> dictates the object allocation rate. Given enough concurrency, >>> even a very small number of object allocations per request can >>> create an object allocation rate that can no longer be sustained. >>> >>> >> I'm saying that the number of concurrent requests might not dictate >> object allocation rate. There are probably a number of allocations >> that >> happen after the EJB instance is obtained. i.e. interception chains, >> contexts, etc. If StrictMaxPool blocks until a new instance is >> available, then there would be less allocations per request as >> blocking >> threads would be serialized. >> >> Whoever is investigating StrictMaxPool, or EJB pooling in general >> should >> stop. Its pointless. >> >> > Ah, no its not pointless. We have a new non-blocking implementation of StrictMaxPool, and its upstream in Wildfly 9, and will be in EAP 6.4. It has helped us increase our throughput, and reduce response times alot! > > Andy > Some contextual numbers around what Andy is describing. These are results from one of our benchmarks; Average response times (28600 concurrent users) Pooled Non-pooled Remote EJB invocations 0.114s 2.094s WS invocations 0.105s 0.332s HTTP web app invocations HttpCallTypeA 0.090s 5.589s HttpCallTypeB 0.042s 2.510s HttpCallTypeC 0.116s 7.267S

From: "Jason Greene" <jason.greene(a)redhat.com&gt; To: "Andrig Miller" <anmiller(a)redhat.com&gt; Cc: "Bill Burke" <bburke(a)redhat.com&gt;, wildfly-dev(a)lists.jboss.org Sent: Tuesday, August 5, 2014 8:15:47 AM Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default On Aug 5, 2014, at 8:43 AM, Andrig Miller <anmiller(a)redhat.com&gt; wrote: > > > ----- Original Message ----- >> From: "Bill Burke" <bburke(a)redhat.com&gt; >> To: "James Livingston" <jlivings(a)redhat.com&gt; >> Cc: wildfly-dev(a)lists.jboss.org >> Sent: Tuesday, August 5, 2014 7:32:04 AM >> Subject: Re: [wildfly-dev] Pooling EJB Session Beans per default >> >> >> >> On 8/4/2014 8:44 PM, James Livingston wrote: >>> On Mon, 2014-08-04 at 20:22 -0400, Bill Burke wrote: >>>> I always liked the ThreadLocal pool. No synchronization, >>>> little >>>> to no >>>> allocations. >>> >>> It also can cause massive memory leaks if invoked from threads >>> which >>> aren't re-used, like timer threads, and precautions aren't taken. >>> AS/EAP >>> 5 suffered from that problem with the default ThreadLocalPool. >>> >> >> Which is also something that could be mitigated by a common thread >> facility. Something AS/EAP 5 didn't have. >> >> Honestly though, I think this talk of EJB pooling is ridiculous. >> Component layers like CDI and JAX-RS don't have pooled >> architectures >> for >> their per-request objects. Also, in an average application, there >> are >> multiple orders of magnitude more non component classes that are >> being >> instantiated per request. Just think of all the Strings created >> by a >> simple HTTP request. >> >> You want a better focus? How about JSON/XML marshalling? Which >> could >> make things much easier to maintain then the hand-coded parsers >> that >> Wildfly uses to parse config. And much faster and less memory at >> runtime for SOAP and JAX-RS request that currently rely on java >> reflection. >> >> You could go research perfect hashing algorithms for URL matching >> with >> servlets and JAX-RS. >> >> You could go do some perf analysis of all of our frameworks to >> make >> memory reduction and speed recommendations or even Pull Requests. >> >> You could visit each major project and make sure our automated >> builds >> have and can run automated stress tests and are measured against >> previous versions. >> >> Or you could just focus on these silly benchmarks that test no-op >> HTTP >> and EJB requests. >> > > Those silly benchmarks, are indeed silly. Any workload that > doesn't actually do anything with the requests is not very > helpful. You can be really fast on them, and be really slow on a > workload that actually does something. We have already seen this > in one case recently. It’s all about precision, and limiting variables. I know its fun to criticize micro benchmarks (obviously micro is indeed relative because a great deal of processing goes on even when you have EJB and HTTP requests that don’t do much), however micro benchmarks, provided they are measured correctly, are quick to run and make it easy to tell what you have to fix. Large gigantic apps on the other hand are difficult to analyze, and do not necessarily represent activity that a differently architected app would show. To use an example, if you have pattern matching code, and you want it to be fast, it’s much easier (and accurate) to benchmark the pattern matching code then it is to benchmaark a full EE app that happens to use pattern matching in a few places.

Arguing against targeted benchmarks is like arguing that you should never write a test case, because only real production load will show problems…. To some extent thats true, but that doesn’t mean that the problems you catch in simulations won’t affect production users.

-- Jason T. Greene WildFly Lead / JBoss EAP Platform Architect JBoss, a division of Red Hat

On Mon, 2014-08-04 at 20:22 -0400, Bill Burke wrote: > I always liked the ThreadLocal pool. No synchronization, little to no > allocations. It also can cause massive memory leaks if invoked from threads which aren't re-used, like timer threads, and precautions aren't taken. AS/EAP 5 suffered from that problem with the default ThreadLocalPool.

The major problem I've seen in support cases related to StrictMaxPool is the very arbitrary default size. I believe it used to be 30, and almost no-one knew that they might need to tune it to fit their applications' usage patterns and workload. Having a rising wait-time metric for the pool indicates that it may be too small, but I believe the statistics are disabled by default for performance reasons.