[infinispan-issues] [JBoss JIRA] (ISPN-6580) Hotrod performance regressions after ISPN-5342 ISPN-6545

[William Burns (JIRA)]

    [ https://issues.jboss.org/browse/ISPN-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201264#comment-13201264 ] 

William Burns commented on ISPN-6580:
-------------------------------------

I tweaked my server to not have the execution group changes and the performance is the same as the old server.  Unfortunately by doing this we cannot support async clients and event OOM will not be solved.  Unfortunately it seems like the context switching caused by handing off the messages is causing a major performance hit for operations that don't do remote operations.

I am thinking to fix this by possibly having a get operation performed on the handler thread instead of handing it off to the execution thread.  Unfortunately this means for an async client the server could read multiple requests but only be able handle 1 remote get instead of possibly multiple in parallel, but other operations would work in parallel.  This might be okay, but if the client has an outdated topology this could cause a major slowdown if it has to go remote.  I am thinking I might want to add an extra handler that will delegate calls if no remote operations are required, possibly.

> Hotrod performance regressions after ISPN-5342 ISPN-6545
> --------------------------------------------------------
>
>                 Key: ISPN-6580
>                 URL: https://issues.jboss.org/browse/ISPN-6580
>             Project: Infinispan
>          Issue Type: Bug
>          Components: Remote Protocols, Server
>            Reporter: Jakub Markos
>            Assignee: William Burns
>         Attachments: jfr_recordings.zip, pom.xml, Reproducer.java
>
>
> There were 2 recent regressions in hotrod performance, one between commits dd5501c5e and 628819461 and the second one between 628819461 and db0890270. I didn't look for the exact commits, so the name of the issue might not be 100% exact...
> It is easily reproducable locally with a single server instance, reproducer attached.
> The numbers on my machine:
> ||Build commit||Puts time||Gets time||
> |dd5501c5e|21|74|
> |628819461|26|102|
> |db0890270|48|224|
> The JFR recordings (attached, captured is only the part of the test with gets) for db0890270 show a lot of time is spent in HotRodDecoder#resetNow(), and also the allocation rate goes from 100MB/s for dd5501c5e to over 1GB/s for db0890270. There are no glaring differences between dd5501c5e and 628819461.



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)