For the JPA tests, I wonder if different persistence.xml settings might be interesting to use. For example, adding "<shared-cache-mode>NONE</shared-cache-mode>" to https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... would disable the second level cache for persistence providers that have a second level cache enabled by default (e.g. https://wiki.eclipse.org/EclipseLink/Examples/JPA/Caching). Perhaps we could also see runs with "<shared-cache-mode>ALL</shared-cache-mode>" via an alternative persistence unit definition. Which would allow a second level cache to be enabled. Perhaps we could also add the various vendor specific hints to enable the second level cache (e.g. <property name="hibernate.cache.use_second_level_cache" value="true"/> and <property name="openjpa.DataCache" value="true"/>)

I'm not sure if there will be any impact for the SingleQuery test but there could be differences for the other JPA tests. On 02/12/2014 12:14 PM, Cody Lerum wrote: > I've had this queued up for a bit but Stuart suggested I hold off > until Wildfly went final so here it is. > > Some of you are probably aware of a Web Framework Benchmark project > that is being put on by TechEmpower > (http://www.techempower.com/benchmarks/). They are currently up to the > 8th round with 90 participating "frameworks". Undertow as been around > for a few rounds and is routinely at or near the top of the charts. > > My goal here is to incorporate EE7 into these results with Wildfly > providing the implementation. There are a lot of different ways to go > here, but my preference is to stay portable with the application so > that if another EE7 implementation wanted to use the same code it > would be an apples to apples comparison. IMO this means for example > that it shouldn't dip down into undertow/hibernate/resteasy/etc > specific features that wouldn't be available on other application > servers. Stick to EE7 API. > > There currently are 6 tests > > 1. Plaintext: Simple "Hello World!" Text response. Not that > interesting and not dynamic. Implemented via JAX-RS. > https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > > 2. JSON Serialization: "Hello World" placed into a new object and then > serialized out to JSON. Again not really a real world thing since > nothing dynamic is happening here. Implemented via JAX-RS. > https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > > 3. Single Query: Select a single random result from a database and > serialize out to JSON. First real world test. Implemented via JAX-RS, > CDI, JPA. https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > > 4. Multiple Queries: Select a specified number of random results from > database one by one. Add to list and serialize out to JSON. > Implemented via JAX-RS, CDI, JPA. > https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > > 5. Data Updates: Select a specifed number of random results from > database one by one and update a value on each of them. Then return > the result via JSON. Implemented via JAX-RS, CDI, JPA. > https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > > 6. Fortunes: Select all from database and store in a list. Add a new > result and sort. Display sorted results as an html table. Implemented > via JSF, CDI, JPA. > https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > > Since 1 and 2 are handled so well by undertow and really aren't > anything someone would do real world with an application server I > could see dropping them and just not participating in those tests. > Unless anyone thinks differently. > > On the other tests it would be helpful if you could review the code to > see if there are better/faster ways to accomplish the same tasks > without dipping into implementation specific features that would not > be portable. > > Thanks! > > -C > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev > _______________________________________________ wildfly-dev mailing list wildfly-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/wildfly-dev

Sorry, I should have pointed out that caching is not permitted in any of the current tests. It may however be allowed in futures tests.

Full restrictions and requirements here - http://www.techempower.com/benchmarks/#section=code On Wed, Feb 12, 2014 at 7:51 PM, Scott Marlow <smarlow(a)redhat.com&gt; wrote: > On 02/12/2014 09:48 PM, Scott Marlow wrote: >> >> For the JPA tests, I wonder if different persistence.xml settings might >> be interesting to use. For example, adding >> "<shared-cache-mode>NONE</shared-cache-mode>" to >> >> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... >> would disable the second level cache for persistence providers that have >> a second level cache enabled by default (e.g. >> https://wiki.eclipse.org/EclipseLink/Examples/JPA/Caching). >> >> Perhaps we could also see runs with >> "<shared-cache-mode>ALL</shared-cache-mode>" via an alternative >> persistence unit definition. Which would allow a second level cache to >> be enabled. Perhaps we could also add the various vendor specific hints >> to enable the second level cache (e.g. <property >> name="hibernate.cache.use_second_level_cache" value="true"/> and >> <property name="openjpa.DataCache" value="true"/>) > > > Should also add @Cacheable(true) to the entities to go with this. > >> >> I'm not sure if there will be any impact for the SingleQuery test but >> there could be differences for the other JPA tests. >> >> On 02/12/2014 12:14 PM, Cody Lerum wrote: >>> >>> I've had this queued up for a bit but Stuart suggested I hold off >>> until Wildfly went final so here it is. >>> >>> Some of you are probably aware of a Web Framework Benchmark project >>> that is being put on by TechEmpower >>> (http://www.techempower.com/benchmarks/). They are currently up to the >>> 8th round with 90 participating "frameworks". Undertow as been around >>> for a few rounds and is routinely at or near the top of the charts. >>> >>> My goal here is to incorporate EE7 into these results with Wildfly >>> providing the implementation. There are a lot of different ways to go >>> here, but my preference is to stay portable with the application so >>> that if another EE7 implementation wanted to use the same code it >>> would be an apples to apples comparison. IMO this means for example >>> that it shouldn't dip down into undertow/hibernate/resteasy/etc >>> specific features that wouldn't be available on other application >>> servers. Stick to EE7 API. >>> >>> There currently are 6 tests >>> >>> 1. Plaintext: Simple "Hello World!" Text response. Not that >>> interesting and not dynamic. Implemented via JAX-RS. >>> >>> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... >>> >>> 2. JSON Serialization: "Hello World" placed into a new object and then >>> serialized out to JSON. Again not really a real world thing since >>> nothing dynamic is happening here. Implemented via JAX-RS. >>> >>> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... >>> >>> 3. Single Query: Select a single random result from a database and >>> serialize out to JSON. First real world test. Implemented via JAX-RS, >>> CDI, JPA. >>> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... >>> >>> 4. Multiple Queries: Select a specified number of random results from >>> database one by one. Add to list and serialize out to JSON. >>> Implemented via JAX-RS, CDI, JPA. >>> >>> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... >>> >>> 5. Data Updates: Select a specifed number of random results from >>> database one by one and update a value on each of them. Then return >>> the result via JSON. Implemented via JAX-RS, CDI, JPA. >>> >>> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... >>> >>> 6. Fortunes: Select all from database and store in a list. Add a new >>> result and sort. Display sorted results as an html table. Implemented >>> via JSF, CDI, JPA. >>> >>> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... >>> >>> Since 1 and 2 are handled so well by undertow and really aren't >>> anything someone would do real world with an application server I >>> could see dropping them and just not participating in those tests. >>> Unless anyone thinks differently. >>> >>> On the other tests it would be helpful if you could review the code to >>> see if there are better/faster ways to accomplish the same tasks >>> without dipping into implementation specific features that would not >>> be portable. >>> >>> Thanks! >>> >>> -C >>> _______________________________________________ >>> wildfly-dev mailing list >>> wildfly-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>> >> >> _______________________________________________ >> wildfly-dev mailing list >> wildfly-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/wildfly-dev >> >

Just to be clear this wouldn't have affected the default hibernate impl in wildfly, but may have allowed caching if deployed on an app server using eclipselink or some other JPA impl correct?

Either way it has been added.

On Thu, Feb 13, 2014 at 4:32 AM, Scott Marlow <smarlow(a)redhat.com&gt; wrote: > On 02/12/2014 10:04 PM, Cody Lerum wrote: >> >> Sorry, I should have pointed out that caching is not permitted in any >> of the current tests. It may however be allowed in futures tests. > > > Should the EclipseLink second level cache (enabled by default), be disabled > then? > > Adding "<shared-cache-mode>NONE</shared-cache-mode>" to the persistence.xml > should help accomplish that. > > >> >> Full restrictions and requirements here - >> http://www.techempower.com/benchmarks/#section=code >> >> On Wed, Feb 12, 2014 at 7:51 PM, Scott Marlow <smarlow(a)redhat.com&gt; wrote: >>> >>> On 02/12/2014 09:48 PM, Scott Marlow wrote: >>>> >>>> >>>> For the JPA tests, I wonder if different persistence.xml settings might >>>> be interesting to use. For example, adding >>>> "<shared-cache-mode>NONE</shared-cache-mode>" to >>>> >>>> >>>> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... >>>> would disable the second level cache for persistence providers that have >>>> a second level cache enabled by default (e.g. >>>> https://wiki.eclipse.org/EclipseLink/Examples/JPA/Caching). >>>> >>>> Perhaps we could also see runs with >>>> "<shared-cache-mode>ALL</shared-cache-mode>" via an alternative >>>> persistence unit definition. Which would allow a second level cache to >>>> be enabled. Perhaps we could also add the various vendor specific hints >>>> to enable the second level cache (e.g. <property >>>> name="hibernate.cache.use_second_level_cache" value="true"/> and >>>> <property name="openjpa.DataCache" value="true"/>) >>> >>> >>> >>> Should also add @Cacheable(true) to the entities to go with this. >>> >>>> >>>> I'm not sure if there will be any impact for the SingleQuery test but >>>> there could be differences for the other JPA tests. >>>> >>>> On 02/12/2014 12:14 PM, Cody Lerum wrote: >>>>> >>>>> >>>>> I've had this queued up for a bit but Stuart suggested I hold off >>>>> until Wildfly went final so here it is. >>>>> >>>>> Some of you are probably aware of a Web Framework Benchmark project >>>>> that is being put on by TechEmpower >>>>> (http://www.techempower.com/benchmarks/). They are currently up to the >>>>> 8th round with 90 participating "frameworks". Undertow as been around >>>>> for a few rounds and is routinely at or near the top of the charts. >>>>> >>>>> My goal here is to incorporate EE7 into these results with Wildfly >>>>> providing the implementation. There are a lot of different ways to go >>>>> here, but my preference is to stay portable with the application so >>>>> that if another EE7 implementation wanted to use the same code it >>>>> would be an apples to apples comparison. IMO this means for example >>>>> that it shouldn't dip down into undertow/hibernate/resteasy/etc >>>>> specific features that wouldn't be available on other application >>>>> servers. Stick to EE7 API. >>>>> >>>>> There currently are 6 tests >>>>> >>>>> 1. Plaintext: Simple "Hello World!" Text response. Not that >>>>> interesting and not dynamic. Implemented via JAX-RS. >>>>> >>>>> >>>>> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... >>>>> >>>>> 2. JSON Serialization: "Hello World" placed into a new object and then >>>>> serialized out to JSON. Again not really a real world thing since >>>>> nothing dynamic is happening here. Implemented via JAX-RS. >>>>> >>>>> >>>>> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... >>>>> >>>>> 3. Single Query: Select a single random result from a database and >>>>> serialize out to JSON. First real world test. Implemented via JAX-RS, >>>>> CDI, JPA. >>>>> >>>>> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... >>>>> >>>>> 4. Multiple Queries: Select a specified number of random results from >>>>> database one by one. Add to list and serialize out to JSON. >>>>> Implemented via JAX-RS, CDI, JPA. >>>>> >>>>> >>>>> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... >>>>> >>>>> 5. Data Updates: Select a specifed number of random results from >>>>> database one by one and update a value on each of them. Then return >>>>> the result via JSON. Implemented via JAX-RS, CDI, JPA. >>>>> >>>>> >>>>> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... >>>>> >>>>> 6. Fortunes: Select all from database and store in a list. Add a new >>>>> result and sort. Display sorted results as an html table. Implemented >>>>> via JSF, CDI, JPA. >>>>> >>>>> >>>>> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... >>>>> >>>>> Since 1 and 2 are handled so well by undertow and really aren't >>>>> anything someone would do real world with an application server I >>>>> could see dropping them and just not participating in those tests. >>>>> Unless anyone thinks differently. >>>>> >>>>> On the other tests it would be helpful if you could review the code to >>>>> see if there are better/faster ways to accomplish the same tasks >>>>> without dipping into implementation specific features that would not >>>>> be portable. >>>>> >>>>> Thanks! >>>>> >>>>> -C >>>>> _______________________________________________ >>>>> wildfly-dev mailing list >>>>> wildfly-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>>> >>>> >>>> _______________________________________________ >>>> wildfly-dev mailing list >>>> wildfly-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev >>>> >>> >

One thing that does worry me here is that people are going to compare JAXRS and CDI performance against resins plain servlet performance. From my testing I know that these do add overhead (and I am going to work on this when I get back from PTO). At the very least we should also have a plain servlet version that is directly comparable. Stuart Sent from my iPhone On 12/02/2014, at 22:44, Cody Lerum <cody.lerum(a)gmail.com&gt; wrote: > I've had this queued up for a bit but Stuart suggested I hold off > until Wildfly went final so here it is. > > Some of you are probably aware of a Web Framework Benchmark project > that is being put on by TechEmpower > (http://www.techempower.com/benchmarks/). They are currently up to the > 8th round with 90 participating "frameworks". Undertow as been around > for a few rounds and is routinely at or near the top of the charts. > > My goal here is to incorporate EE7 into these results with Wildfly > providing the implementation. There are a lot of different ways to go > here, but my preference is to stay portable with the application so > that if another EE7 implementation wanted to use the same code it > would be an apples to apples comparison. IMO this means for example > that it shouldn't dip down into undertow/hibernate/resteasy/etc > specific features that wouldn't be available on other application > servers. Stick to EE7 API. > > There currently are 6 tests > > 1. Plaintext: Simple "Hello World!" Text response. Not that > interesting and not dynamic. Implemented via JAX-RS. > https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > > 2. JSON Serialization: "Hello World" placed into a new object and then > serialized out to JSON. Again not really a real world thing since > nothing dynamic is happening here. Implemented via JAX-RS. > https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > > 3. Single Query: Select a single random result from a database and > serialize out to JSON. First real world test. Implemented via JAX-RS, > CDI, JPA. https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > > 4. Multiple Queries: Select a specified number of random results from > database one by one. Add to list and serialize out to JSON. > Implemented via JAX-RS, CDI, JPA. > https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > > 5. Data Updates: Select a specifed number of random results from > database one by one and update a value on each of them. Then return > the result via JSON. Implemented via JAX-RS, CDI, JPA. > https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > > 6. Fortunes: Select all from database and store in a list. Add a new > result and sort. Display sorted results as an html table. Implemented > via JSF, CDI, JPA. > https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > > Since 1 and 2 are handled so well by undertow and really aren't > anything someone would do real world with an application server I > could see dropping them and just not participating in those tests. > Unless anyone thinks differently. > > On the other tests it would be helpful if you could review the code to > see if there are better/faster ways to accomplish the same tasks > without dipping into implementation specific features that would not > be portable. > > Thanks! > > -C > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev

Cody, as we spoke on IRC few weeks back, lets do optimized version of plaintext & json tests using ServletExtension. I can help you with that in coming days, just to clear few things I got piled up. -- tomaz On Thu, Feb 13, 2014 at 4:40 PM, Cody Lerum <cody.lerum(a)gmail.com&gt; wrote: > > Stuart, > > That does worry me as well. The benchmarks can be misread if not > comparing apples to apples and properly applying filters. > > Raw DB vs ORM for example is about a 2x difference. > (http://www.techempower.com/benchmarks/#section=data-r8&hw=i7&test... > vs > http://www.techempower.com/benchmarks/#section=data-r8&hw=i7&test...) > > Someone could for example look at > http://www.techempower.com/benchmarks/#section=data-r8&hw=ec2&tes... > and determine that undertow was the "fastest", but we all know it > would be a vastly different experience writing and maintaining a full > application on pure undertow vs something like say ee7. > > For example the single query test. > > > https://github.com/codylerum/FrameworkBenchmarks/blob/master/undertow/src... > > vs > > > https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > > The the EE7 code is very minimal and concise because so much is > provided by the framework, but that obviously is going to come at a > cost in terms of performance. > > Anyway I would imagine there would eventually be a few permutations of > the tests utilizing wildfly as the application server itself can be > used in so many different ways. > > wildfly-ee7 (CDI/JAX-RS/JPA) > > wildfly-servlet > > Maybe someone could look at reusing or adapting the existing servlet > test to run on wildfly instead of resin? - > https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/servlet > > -C > > On Thu, Feb 13, 2014 at 2:11 AM, Stuart Douglas > <stuart.w.douglas(a)gmail.com&gt; wrote: > > One thing that does worry me here is that people are going to compare > > JAXRS and CDI performance against resins plain servlet performance. From my > > testing I know that these do add overhead (and I am going to work on this > > when I get back from PTO). > > > > At the very least we should also have a plain servlet version that is > > directly comparable. > > > > Stuart > > > > Sent from my iPhone > > > > On 12/02/2014, at 22:44, Cody Lerum <cody.lerum(a)gmail.com&gt; wrote: > > > >> I've had this queued up for a bit but Stuart suggested I hold off > >> until Wildfly went final so here it is. > >> > >> Some of you are probably aware of a Web Framework Benchmark project > >> that is being put on by TechEmpower > >> (http://www.techempower.com/benchmarks/). They are currently up to the > >> 8th round with 90 participating "frameworks". Undertow as been around > >> for a few rounds and is routinely at or near the top of the charts. > >> > >> My goal here is to incorporate EE7 into these results with Wildfly > >> providing the implementation. There are a lot of different ways to go > >> here, but my preference is to stay portable with the application so > >> that if another EE7 implementation wanted to use the same code it > >> would be an apples to apples comparison. IMO this means for example > >> that it shouldn't dip down into undertow/hibernate/resteasy/etc > >> specific features that wouldn't be available on other application > >> servers. Stick to EE7 API. > >> > >> There currently are 6 tests > >> > >> 1. Plaintext: Simple "Hello World!" Text response. Not that > >> interesting and not dynamic. Implemented via JAX-RS. > >> > >> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > >> > >> 2. JSON Serialization: "Hello World" placed into a new object and then > >> serialized out to JSON. Again not really a real world thing since > >> nothing dynamic is happening here. Implemented via JAX-RS. > >> > >> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > >> > >> 3. Single Query: Select a single random result from a database and > >> serialize out to JSON. First real world test. Implemented via JAX-RS, > >> CDI, JPA. > >> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > >> > >> 4. Multiple Queries: Select a specified number of random results from > >> database one by one. Add to list and serialize out to JSON. > >> Implemented via JAX-RS, CDI, JPA. > >> > >> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > >> > >> 5. Data Updates: Select a specifed number of random results from > >> database one by one and update a value on each of them. Then return > >> the result via JSON. Implemented via JAX-RS, CDI, JPA. > >> > >> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > >> > >> 6. Fortunes: Select all from database and store in a list. Add a new > >> result and sort. Display sorted results as an html table. Implemented > >> via JSF, CDI, JPA. > >> > >> https://github.com/codylerum/FrameworkBenchmarks/blob/ee7-wildfly/ee7-wil... > >> > >> Since 1 and 2 are handled so well by undertow and really aren't > >> anything someone would do real world with an application server I > >> could see dropping them and just not participating in those tests. > >> Unless anyone thinks differently. > >> > >> On the other tests it would be helpful if you could review the code to > >> see if there are better/faster ways to accomplish the same tasks > >> without dipping into implementation specific features that would not > >> be portable. > >> > >> Thanks! > >> > >> -C > >> _______________________________________________ > >> wildfly-dev mailing list > >> wildfly-dev(a)lists.jboss.org > >> https://lists.jboss.org/mailman/listinfo/wildfly-dev > _______________________________________________ > wildfly-dev mailing list > wildfly-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/wildfly-dev