[JBoss JIRA] (WFLY-3858) Infinispan cache configuration is not always applied to security-domain
by xiaodong xie (JIRA)
[ https://issues.jboss.org/browse/WFLY-3858?page=com.atlassian.jira.plugin.... ]
xiaodong xie edited comment on WFLY-3858 at 5/18/16 12:47 PM:
--------------------------------------------------------------
The workaround I found (and I am using Wildfly-10), was to use this method when interacting with the cache: 'V put(K key, V value, long lifespan, TimeUnit lifespanUnit, long maxIdleTime, TimeUnit maxIdleTimeUnit);'. And I verified that this works.
What's missing is that, we cannot change the eviction strategy from NONE to anything else, as that setting is immutable... And unfortunately, based on my verification, the configuration change with "cacheConfiguration.eviction().size(XXX);" is not respected either... So we cannot set a cache size limit...
Some discussions here: https://developer.jboss.org/thread/267907?start=0&tstart=0
was (Author: xxd):
The workaround I found (and I am using Wildfly-10), was to use this method when interacting with the cache: 'V put(K key, V value, long lifespan, TimeUnit lifespanUnit, long maxIdleTime, TimeUnit maxIdleTimeUnit);'. And I verified that this works.
What's missing is that, we cannot change the eviction strategy from NONE to anything else, as that setting is immutable... But fortunately, based on my verification, the configuration change with "cacheConfiguration.eviction().size(XXX);" is respected... So we still are able to set a cache size limit...
Some discussions here: https://developer.jboss.org/thread/267907?start=0&tstart=0
> Infinispan cache configuration is not always applied to security-domain
> -----------------------------------------------------------------------
>
> Key: WFLY-3858
> URL: https://issues.jboss.org/browse/WFLY-3858
> Project: WildFly
> Issue Type: Bug
> Components: Security
> Affects Versions: 8.1.0.Final
> Reporter: Robert Tuck
> Assignee: Darran Lofthouse
> Attachments: debugger output.txt
>
>
> On Wildfly 8.1.0.Final, I have the following standalone-ha.xml:
> <subsystem xmlns="urn:jboss:domain:infinispan:2.0">
> ...
> <cache-container name="security" default-cache="auth-cache" start="EAGER">
> <transport cluster="${cluster.name}_SEC" lock-timeout="60000"/>
> <replicated-cache name="auth-cache" batching="true" mode="ASYNC">
> <eviction strategy="LRU" max-entries="10000"/>
> <expiration lifespan="60000"/>
> </replicated-cache>
> </cache-container>
> </subsystem>
> ...
> <subsystem xmlns="urn:jboss:domain:security:1.2">
> <security-domains>
> ...
> <security-domain name="OAuth-Consumer" cache-type="infinispan">
> <authentication>
> <login-module code="com.idbs.ewb.server.auth.module.OAuthConsumerLoginModule" flag="sufficient"
> module="deployment.ewb-server-ear.ear">
> <module-option name="allowedConsumerAuthFailures" value="-1"/>
> <module-option name="consumerLoginFailureTimeoutMs" value="3000"/>
> </login-module>
> </authentication>
> </security-domain>
> </security-domains>
> </subsystem>
> After startup the OAuth-Consumer security domain cache "auth-cache" is not always configured with the specified settings (~50% of the time). This can be verified by monitoring the jboss.infinispan nodes with JConsole and retrieving the cache settings, and tracking the cache hits during logins. This shows that succesful logins are cached but do not expire after the expected 60s, and that the expiration lifespan is set to -1 rather than 60000, as are eviction max entries.
> After some debugging I have narrowed down the problem to some code that gets called from inside org.jboss.as.security.plugins.JNDIBasedSecurityManagement:
> public SecurityDomainContext createSecurityDomainContext(String securityDomain, Object cacheFactory) throws Exception {
> log.debugf("Creating SDC for domain=" + securityDomain);
> AuthenticationManager am = createAuthenticationManager(securityDomain);
> // create authentication cache
> if (cacheFactory instanceof EmbeddedCacheManager) {
> EmbeddedCacheManager cacheManager = EmbeddedCacheManager.class.cast(cacheFactory);
> @SuppressWarnings("rawtypes")
> Cache cache = null;
> if (cacheManager != null) {
> // TODO override global settings with security domain specific
> ConfigurationBuilder builder = new ConfigurationBuilder();
> Configuration baseCfg = cacheManager.getCacheConfiguration("auth-cache");
> ^^^^ This call here doesn’t always return the correct configuration, baseCfg is then null.
> if (baseCfg != null) {
> builder.read(baseCfg);
> }
> cacheManager.defineConfiguration(securityDomain, builder.build());
> cache = cacheManager.getCache(securityDomain);
> }
> if (cache != null && am instanceof CacheableManager) {
> @SuppressWarnings({ "unchecked", "rawtypes" })
> CacheableManager<Map, Principal> cm = (CacheableManager<Map, Principal>) am;
> cm.setCache(cache);
> }
> } else if (cacheFactory instanceof DefaultAuthenticationCacheFactory) {
> <…>
> }
> getCacheConfiguration(String) is implemented inside org.infinispan.manager.DefaultCacheManager:
> @Override
> public Configuration getCacheConfiguration(String name) {
> Configuration configuration = configurationOverrides.get(name);
> if (configuration == null && cacheExists(name)) {
> return defaultConfiguration;
> }
> return configuration;
> }
> Seems like the condition configuration == null occurs when the cache doesn’t exist, therefore it returns null. This appears to be a race condition between this code and where it gets registered in wireAndStartCache(String).
>
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 11 months
[JBoss JIRA] (JBASMP-77) ConcurrentModificationException in deploy-artifact
by James Perkins (JIRA)
[ https://issues.jboss.org/browse/JBASMP-77?page=com.atlassian.jira.plugin.... ]
James Perkins reassigned JBASMP-77:
-----------------------------------
Assignee: James Perkins
> ConcurrentModificationException in deploy-artifact
> --------------------------------------------------
>
> Key: JBASMP-77
> URL: https://issues.jboss.org/browse/JBASMP-77
> Project: JBoss AS Maven Plugins
> Issue Type: Bug
> Components: deploy
> Affects Versions: 7.7.Final
> Reporter: Daniele Gaffuri
> Assignee: James Perkins
> Fix For: 7.8.Final
>
>
> In the same situation of JBASMP-57 deploy-artifact throws the following exception:
> {code}
> [ERROR] Failed to execute goal org.jboss.as.plugins:jboss-as-maven-plugin:7.7.SCE:deploy-artifact (deploy) on project sideup-integration: Could not execute goal deploy-artifact on null. Reason: null: ConcurrentModificationException -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.jboss.as.plugins:jboss-as-maven-plugin:7.7.SCE:deploy-artifact (deploy) on project sideup-integration: Could not execute goal deploy-artifact on null. Reason: null
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
> at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
> at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
> at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
> Caused by: org.apache.maven.plugin.MojoExecutionException: Could not execute goal deploy-artifact on null. Reason: null
> at org.jboss.as.plugin.deployment.AbstractDeployment.doExecute(AbstractDeployment.java:159)
> at org.jboss.as.plugin.deployment.AbstractDeployment.execute(AbstractDeployment.java:112)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
> ... 19 more
> Caused by: java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)
> at java.util.LinkedHashMap$KeyIterator.next(LinkedHashMap.java:405)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:341)
> at org.jboss.as.plugin.deployment.DeployArtifact.validate(DeployArtifact.java:84)
> at org.jboss.as.plugin.deployment.AbstractDeployment.doExecute(AbstractDeployment.java:136)
> ... 22 more
> {code}
> This is probably due to changes in related commit [37bc2c1], causing both deploy and undeploy validate methods to add the set of dependencies to itself:
> {code}
> final Set<Artifact> dependencies = project.getDependencyArtifacts();
> // Allows provided dependencies to be seen
> dependencies.addAll(project.getDependencyArtifacts());
> {code}
> {code}
> final Set<Artifact> dependencies = project.getDependencyArtifacts();
> dependencies.addAll(this.project.getDependencyArtifacts());
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 11 months
[JBoss JIRA] (JBASMP-77) ConcurrentModificationException in deploy-artifact
by James Perkins (JIRA)
[ https://issues.jboss.org/browse/JBASMP-77?page=com.atlassian.jira.plugin.... ]
James Perkins reassigned JBASMP-77:
-----------------------------------
Assignee: Tomas Remes (was: James Perkins)
> ConcurrentModificationException in deploy-artifact
> --------------------------------------------------
>
> Key: JBASMP-77
> URL: https://issues.jboss.org/browse/JBASMP-77
> Project: JBoss AS Maven Plugins
> Issue Type: Bug
> Components: deploy
> Affects Versions: 7.7.Final
> Reporter: Daniele Gaffuri
> Assignee: Tomas Remes
> Fix For: 7.8.Final
>
>
> In the same situation of JBASMP-57 deploy-artifact throws the following exception:
> {code}
> [ERROR] Failed to execute goal org.jboss.as.plugins:jboss-as-maven-plugin:7.7.SCE:deploy-artifact (deploy) on project sideup-integration: Could not execute goal deploy-artifact on null. Reason: null: ConcurrentModificationException -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.jboss.as.plugins:jboss-as-maven-plugin:7.7.SCE:deploy-artifact (deploy) on project sideup-integration: Could not execute goal deploy-artifact on null. Reason: null
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:217)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
> at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
> at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
> at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290)
> at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230)
> at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)
> at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)
> Caused by: org.apache.maven.plugin.MojoExecutionException: Could not execute goal deploy-artifact on null. Reason: null
> at org.jboss.as.plugin.deployment.AbstractDeployment.doExecute(AbstractDeployment.java:159)
> at org.jboss.as.plugin.deployment.AbstractDeployment.execute(AbstractDeployment.java:112)
> at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
> ... 19 more
> Caused by: java.util.ConcurrentModificationException
> at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)
> at java.util.LinkedHashMap$KeyIterator.next(LinkedHashMap.java:405)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:341)
> at org.jboss.as.plugin.deployment.DeployArtifact.validate(DeployArtifact.java:84)
> at org.jboss.as.plugin.deployment.AbstractDeployment.doExecute(AbstractDeployment.java:136)
> ... 22 more
> {code}
> This is probably due to changes in related commit [37bc2c1], causing both deploy and undeploy validate methods to add the set of dependencies to itself:
> {code}
> final Set<Artifact> dependencies = project.getDependencyArtifacts();
> // Allows provided dependencies to be seen
> dependencies.addAll(project.getDependencyArtifacts());
> {code}
> {code}
> final Set<Artifact> dependencies = project.getDependencyArtifacts();
> dependencies.addAll(this.project.getDependencyArtifacts());
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 11 months
[JBoss JIRA] (WFLY-3858) Infinispan cache configuration is not always applied to security-domain
by xiaodong xie (JIRA)
[ https://issues.jboss.org/browse/WFLY-3858?page=com.atlassian.jira.plugin.... ]
xiaodong xie commented on WFLY-3858:
------------------------------------
The workaround I found (and I am using Wildfly-10), was to use this method when interacting with the cache: 'V put(K key, V value, long lifespan, TimeUnit lifespanUnit, long maxIdleTime, TimeUnit maxIdleTimeUnit);'. And I verified that this works.
What's missing is that, we cannot change the eviction strategy from NONE to anything else, as that setting is immutable... But fortunately, based on my verification, the configuration change with "cacheConfiguration.eviction().size(XXX);" is respected... So we still are able to set a cache size limit...
Some discussions here: https://developer.jboss.org/thread/267907?start=0&tstart=0
> Infinispan cache configuration is not always applied to security-domain
> -----------------------------------------------------------------------
>
> Key: WFLY-3858
> URL: https://issues.jboss.org/browse/WFLY-3858
> Project: WildFly
> Issue Type: Bug
> Components: Security
> Affects Versions: 8.1.0.Final
> Reporter: Robert Tuck
> Assignee: Darran Lofthouse
> Attachments: debugger output.txt
>
>
> On Wildfly 8.1.0.Final, I have the following standalone-ha.xml:
> <subsystem xmlns="urn:jboss:domain:infinispan:2.0">
> ...
> <cache-container name="security" default-cache="auth-cache" start="EAGER">
> <transport cluster="${cluster.name}_SEC" lock-timeout="60000"/>
> <replicated-cache name="auth-cache" batching="true" mode="ASYNC">
> <eviction strategy="LRU" max-entries="10000"/>
> <expiration lifespan="60000"/>
> </replicated-cache>
> </cache-container>
> </subsystem>
> ...
> <subsystem xmlns="urn:jboss:domain:security:1.2">
> <security-domains>
> ...
> <security-domain name="OAuth-Consumer" cache-type="infinispan">
> <authentication>
> <login-module code="com.idbs.ewb.server.auth.module.OAuthConsumerLoginModule" flag="sufficient"
> module="deployment.ewb-server-ear.ear">
> <module-option name="allowedConsumerAuthFailures" value="-1"/>
> <module-option name="consumerLoginFailureTimeoutMs" value="3000"/>
> </login-module>
> </authentication>
> </security-domain>
> </security-domains>
> </subsystem>
> After startup the OAuth-Consumer security domain cache "auth-cache" is not always configured with the specified settings (~50% of the time). This can be verified by monitoring the jboss.infinispan nodes with JConsole and retrieving the cache settings, and tracking the cache hits during logins. This shows that succesful logins are cached but do not expire after the expected 60s, and that the expiration lifespan is set to -1 rather than 60000, as are eviction max entries.
> After some debugging I have narrowed down the problem to some code that gets called from inside org.jboss.as.security.plugins.JNDIBasedSecurityManagement:
> public SecurityDomainContext createSecurityDomainContext(String securityDomain, Object cacheFactory) throws Exception {
> log.debugf("Creating SDC for domain=" + securityDomain);
> AuthenticationManager am = createAuthenticationManager(securityDomain);
> // create authentication cache
> if (cacheFactory instanceof EmbeddedCacheManager) {
> EmbeddedCacheManager cacheManager = EmbeddedCacheManager.class.cast(cacheFactory);
> @SuppressWarnings("rawtypes")
> Cache cache = null;
> if (cacheManager != null) {
> // TODO override global settings with security domain specific
> ConfigurationBuilder builder = new ConfigurationBuilder();
> Configuration baseCfg = cacheManager.getCacheConfiguration("auth-cache");
> ^^^^ This call here doesn’t always return the correct configuration, baseCfg is then null.
> if (baseCfg != null) {
> builder.read(baseCfg);
> }
> cacheManager.defineConfiguration(securityDomain, builder.build());
> cache = cacheManager.getCache(securityDomain);
> }
> if (cache != null && am instanceof CacheableManager) {
> @SuppressWarnings({ "unchecked", "rawtypes" })
> CacheableManager<Map, Principal> cm = (CacheableManager<Map, Principal>) am;
> cm.setCache(cache);
> }
> } else if (cacheFactory instanceof DefaultAuthenticationCacheFactory) {
> <…>
> }
> getCacheConfiguration(String) is implemented inside org.infinispan.manager.DefaultCacheManager:
> @Override
> public Configuration getCacheConfiguration(String name) {
> Configuration configuration = configurationOverrides.get(name);
> if (configuration == null && cacheExists(name)) {
> return defaultConfiguration;
> }
> return configuration;
> }
> Seems like the condition configuration == null occurs when the cache doesn’t exist, therefore it returns null. This appears to be a race condition between this code and where it gets registered in wireAndStartCache(String).
>
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 11 months
[JBoss JIRA] (JGRP-2065) RoundTrip: latency is high compared to RoundTripTcp/RoundTripServer
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2065?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-2065 at 5/18/16 12:10 PM:
----------------------------------------------------------
Findings:
* Disabling the thread pools in TCP brought the round-trip time from *110 us* down to *99 us*:
{code:java}
TP transport=channel.getProtocolStack().getTransport();
// uncomment below to disable the regular and OOB thread pools
transport.setOOBThreadPool(new DirectExecutor());
transport.setDefaultThreadPool(new DirectExecutor());
{code}
* Replacing the {{TransferQueueBundler}} with {{NoBundler}} or {{SenderSendsBundler}}: *99 us* down to *69 us*:
{noformat}
probe.sh op=TCP.bundler["no-bundler"]
{noformat}
* Using {{NoBundler}} or {{SenderSendsBundler}} with *2 sender threads*: *69 us* -> *60 us*
* This clearly shows that threading and the bundlers are the biggest latency culprits (so far)
* Remove all protocols at runtime except for {{UNICAST3}}: *no impact*:
{noformat}
probe.sh remove-protocol=MFC
probe.sh print-protocols
{noformat}
* Remove {{UNICAST3}}: *60 us* -> *55 us*
** But at the expense of ordering and retransmission! E.g. if the receiver's pool is full, a message (even though reliably transmitted via TCP/IP) will get discarded and needs to get retransmitted by UNICAST3
was (Author: belaban):
Findings:
* Disabling the thread pools in TCP brought the round-trip time from *110 us* down to *99 us*:
{code:java}
TP transport=channel.getProtocolStack().getTransport();
// uncomment below to disable the regular and OOB thread pools
transport.setOOBThreadPool(new DirectExecutor());
transport.setDefaultThreadPool(new DirectExecutor());
{code}
* Replacing the {{TransferQueueBundler}} with {{NoBundler}} or {{SenderSendsBundler}}: *99 us* down to *69 us*:
{noformat}
probe.sh op=TCP.bundler["no-bundler"]
{noformat}
* Using {{NoBundler}} or {{SenderSendsBundler}} with *2 sender threads*: *69 us* -> *60 us*
* This clearly shows that threading and the bundlers are the biggest latency culprits (so far)
> RoundTrip: latency is high compared to RoundTripTcp/RoundTripServer
> -------------------------------------------------------------------
>
> Key: JGRP-2065
> URL: https://issues.jboss.org/browse/JGRP-2065
> Project: JGroups
> Issue Type: Task
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 4.0
>
>
> {{RoundTrip}} is a simple test between 2 members measuring round-trip latency. The sender continually sends a message, the receiver receives it and sends the response, and the sender unblocks when the response is received. Then the sender sends the next message.
> The time for the request is then logged at the sender and a min/avg/max value is computed (probably changed to histograms later).
> {{RoundTrip}} uses a JGroups channel, configured with {{-props}}, e.g. {{-props ~/tcp.xml}}.
> {{RoundTripTcp}} does the same, but uses direct TCP sockets (no JGroups) for communication.
> {{RoundTripServer}} uses the client-server classes of JGroups for communication, but no channel is used.
> Round trip times (both processes on the same box, a Mac mini):
> * {{RoundTrip}} (with {{tcp.xml}} shipped with JGroups): *110 us*
> * {{RoundTripTcp}}: *46 us*
> * {{RoundTripServer}}: *49 us*
> Note that the client/server classes used by {{RoundTripServer}} are also used by the TCP transport (configured in {{tcp.xml}}.
> {{RoundTripServer}} is ~6% slower than {{RoundTripTcp}}, but that can be attributed to the additional work the former has to do (e.g. connection reaping etc). This is something we can focus on later.
> The big difference are the 110 us for {{RoundTrip}}. The goal is to find out what causes this and eliminate it. Since {{RoundTrip}} and {{RoundTripServer}} use the same underlying client/server classes in JGroups, let's compare these 2.
> Tasks:
> * Remove all protocols other than TCP in the running stack: (e.g. {{probe.sh remove-protocol=MFC}}). I already did this and the diff was negligible, but let's run this again
> * Try various bundlers (e.g. NoBundler)
> * Reduce threads in the threadpools, possibly disable (regular and OOB) thread pools (replace with DirectExecutor)
> * The default is 1 sender thread, but try with multiple threads
> * Measure perf between sending a message (in the bundler) and receiving it
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 11 months
[JBoss JIRA] (JGRP-2065) RoundTrip: latency is high compared to RoundTripTcp/RoundTripServer
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2065?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-2065 at 5/18/16 12:00 PM:
----------------------------------------------------------
Findings:
* Disabling the thread pools in TCP brought the round-trip time from *110 us* down to *99 us*:
{code:java}
TP transport=channel.getProtocolStack().getTransport();
// uncomment below to disable the regular and OOB thread pools
transport.setOOBThreadPool(new DirectExecutor());
transport.setDefaultThreadPool(new DirectExecutor());
{code}
* Replacing the {{TransferQueueBundler}} with {{NoBundler}} or {{SenderSendsBundler}}: *99 us* down to *69 us*:
{noformat}
probe.sh op=TCP.bundler["no-bundler"]
{noformat}
* Using {{NoBundler}} or {{SenderSendsBundler}} with *2 sender threads*: *69 us* -> *60 us*
* This clearly shows that threading and the bundlers are the biggest latency culprits (so far)
was (Author: belaban):
Findings:
* Disabling the thread pools in TCP brought the round-trip time from *110 us* down to *99 us*:
{code:java}
TP transport=channel.getProtocolStack().getTransport();
// uncomment below to disable the regular and OOB thread pools
transport.setOOBThreadPool(new DirectExecutor());
transport.setDefaultThreadPool(new DirectExecutor());
{code}
* Replacing the {{TransferQueueBundler}} with {{NoBundler}} or {{sender-sends}}: *99 us* down to *69 us*:
{noformat}
probe.sh op=TCP.bundler["no-bundler"]
{noformat}
* This clearly shows that threading and the bundlers are the biggest latency culprits (so far)
> RoundTrip: latency is high compared to RoundTripTcp/RoundTripServer
> -------------------------------------------------------------------
>
> Key: JGRP-2065
> URL: https://issues.jboss.org/browse/JGRP-2065
> Project: JGroups
> Issue Type: Task
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 4.0
>
>
> {{RoundTrip}} is a simple test between 2 members measuring round-trip latency. The sender continually sends a message, the receiver receives it and sends the response, and the sender unblocks when the response is received. Then the sender sends the next message.
> The time for the request is then logged at the sender and a min/avg/max value is computed (probably changed to histograms later).
> {{RoundTrip}} uses a JGroups channel, configured with {{-props}}, e.g. {{-props ~/tcp.xml}}.
> {{RoundTripTcp}} does the same, but uses direct TCP sockets (no JGroups) for communication.
> {{RoundTripServer}} uses the client-server classes of JGroups for communication, but no channel is used.
> Round trip times (both processes on the same box, a Mac mini):
> * {{RoundTrip}} (with {{tcp.xml}} shipped with JGroups): *110 us*
> * {{RoundTripTcp}}: *46 us*
> * {{RoundTripServer}}: *49 us*
> Note that the client/server classes used by {{RoundTripServer}} are also used by the TCP transport (configured in {{tcp.xml}}.
> {{RoundTripServer}} is ~6% slower than {{RoundTripTcp}}, but that can be attributed to the additional work the former has to do (e.g. connection reaping etc). This is something we can focus on later.
> The big difference are the 110 us for {{RoundTrip}}. The goal is to find out what causes this and eliminate it. Since {{RoundTrip}} and {{RoundTripServer}} use the same underlying client/server classes in JGroups, let's compare these 2.
> Tasks:
> * Remove all protocols other than TCP in the running stack: (e.g. {{probe.sh remove-protocol=MFC}}). I already did this and the diff was negligible, but let's run this again
> * Try various bundlers (e.g. NoBundler)
> * Reduce threads in the threadpools, possibly disable (regular and OOB) thread pools (replace with DirectExecutor)
> * The default is 1 sender thread, but try with multiple threads
> * Measure perf between sending a message (in the bundler) and receiving it
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 11 months
[JBoss JIRA] (JGRP-2065) RoundTrip: latency is high compared to RoundTripTcp/RoundTripServer
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2065?page=com.atlassian.jira.plugin.... ]
Bela Ban edited comment on JGRP-2065 at 5/18/16 11:52 AM:
----------------------------------------------------------
Findings:
* Disabling the thread pools in TCP brought the round-trip time from *110 us* down to *99 us*:
{code:java}
TP transport=channel.getProtocolStack().getTransport();
// uncomment below to disable the regular and OOB thread pools
transport.setOOBThreadPool(new DirectExecutor());
transport.setDefaultThreadPool(new DirectExecutor());
{code}
* Replacing the {{TransferQueueBundler}} with {{NoBundler}} or {{sender-sends}}: *99 us* down to *69 us*:
{noformat}
probe.sh op=TCP.bundler["no-bundler"]
{noformat}
* This clearly shows that threading and the bundlers are the biggest latency culprits (so far)
was (Author: belaban):
Findings:
* Disabling the thread pools in TCP brought the round-trip time from *110 us* down to *99 us*:
{code:java}
TP transport=channel.getProtocolStack().getTransport();
// uncomment below to disable the regular and OOB thread pools
transport.setOOBThreadPool(new DirectExecutor());
transport.setDefaultThreadPool(new DirectExecutor());
{code}
* Replacing the {{TransferQueueBundler}} with {{NoBundler}}: *99 us* down to *69 us*:
{noformat}
probe.sh op=TCP.bundler["no-bundler"]
{noformat}
* This clearly shows that threading and the bundlers are the biggest latency culprits (so far)
> RoundTrip: latency is high compared to RoundTripTcp/RoundTripServer
> -------------------------------------------------------------------
>
> Key: JGRP-2065
> URL: https://issues.jboss.org/browse/JGRP-2065
> Project: JGroups
> Issue Type: Task
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 4.0
>
>
> {{RoundTrip}} is a simple test between 2 members measuring round-trip latency. The sender continually sends a message, the receiver receives it and sends the response, and the sender unblocks when the response is received. Then the sender sends the next message.
> The time for the request is then logged at the sender and a min/avg/max value is computed (probably changed to histograms later).
> {{RoundTrip}} uses a JGroups channel, configured with {{-props}}, e.g. {{-props ~/tcp.xml}}.
> {{RoundTripTcp}} does the same, but uses direct TCP sockets (no JGroups) for communication.
> {{RoundTripServer}} uses the client-server classes of JGroups for communication, but no channel is used.
> Round trip times (both processes on the same box, a Mac mini):
> * {{RoundTrip}} (with {{tcp.xml}} shipped with JGroups): *110 us*
> * {{RoundTripTcp}}: *46 us*
> * {{RoundTripServer}}: *49 us*
> Note that the client/server classes used by {{RoundTripServer}} are also used by the TCP transport (configured in {{tcp.xml}}.
> {{RoundTripServer}} is ~6% slower than {{RoundTripTcp}}, but that can be attributed to the additional work the former has to do (e.g. connection reaping etc). This is something we can focus on later.
> The big difference are the 110 us for {{RoundTrip}}. The goal is to find out what causes this and eliminate it. Since {{RoundTrip}} and {{RoundTripServer}} use the same underlying client/server classes in JGroups, let's compare these 2.
> Tasks:
> * Remove all protocols other than TCP in the running stack: (e.g. {{probe.sh remove-protocol=MFC}}). I already did this and the diff was negligible, but let's run this again
> * Try various bundlers (e.g. NoBundler)
> * Reduce threads in the threadpools, possibly disable (regular and OOB) thread pools (replace with DirectExecutor)
> * The default is 1 sender thread, but try with multiple threads
> * Measure perf between sending a message (in the bundler) and receiving it
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 11 months
[JBoss JIRA] (JGRP-2065) RoundTrip: latency is high compared to RoundTripTcp/RoundTripServer
by Bela Ban (JIRA)
[ https://issues.jboss.org/browse/JGRP-2065?page=com.atlassian.jira.plugin.... ]
Bela Ban updated JGRP-2065:
---------------------------
Description:
{{RoundTrip}} is a simple test between 2 members measuring round-trip latency. The sender continually sends a message, the receiver receives it and sends the response, and the sender unblocks when the response is received. Then the sender sends the next message.
The time for the request is then logged at the sender and a min/avg/max value is computed (probably changed to histograms later).
{{RoundTrip}} uses a JGroups channel, configured with {{-props}}, e.g. {{-props ~/tcp.xml}}.
{{RoundTripTcp}} does the same, but uses direct TCP sockets (no JGroups) for communication.
{{RoundTripServer}} uses the client-server classes of JGroups for communication, but no channel is used.
Round trip times (both processes on the same box, a Mac mini):
* {{RoundTrip}} (with {{tcp.xml}} shipped with JGroups): *110 us*
* {{RoundTripTcp}}: *46 us*
* {{RoundTripServer}}: *49 us*
Note that the client/server classes used by {{RoundTripServer}} are also used by the TCP transport (configured in {{tcp.xml}}.
{{RoundTripServer}} is ~6% slower than {{RoundTripTcp}}, but that can be attributed to the additional work the former has to do (e.g. connection reaping etc). This is something we can focus on later.
The big difference are the 110 us for {{RoundTrip}}. The goal is to find out what causes this and eliminate it. Since {{RoundTrip}} and {{RoundTripServer}} use the same underlying client/server classes in JGroups, let's compare these 2.
Tasks:
* Remove all protocols other than TCP in the running stack: (e.g. {{probe.sh remove-protocol=MFC}}). I already did this and the diff was negligible, but let's run this again
* Try various bundlers (e.g. NoBundler)
* Reduce threads in the threadpools, possibly disable (regular and OOB) thread pools (replace with DirectExecutor)
* The default is 1 sender thread, but try with multiple threads
* Measure perf between sending a message (in the bundler) and receiving it
was:
{{RoundTrip}} is a simple test between 2 members measuring round-trip latency. The sender continually sends a message, the receiver receives it and sends the response, and the sender unblocks when the response is received. Then the sender sends the next message.
The time for the request is then logged at the sender and a min/avg/max value is computed (probably changed to histograms later).
{{RoundTrip}} uses a JGroups channel, configured with {{-props}}, e.g. {{-props ~/tcp.xml}}.
{{RoundTripTcp}} does the same, but uses direct TCP sockets (no JGroups) for communication.
{{RoundTripServer}} uses the client-server classes of JGroups for communication, but no channel is used.
Round trip times (both processes on the same box, a Mac mini):
* {{RoundTrip}} (with {{tcp.xml}} shipped with JGroups): *110 us*
* {{RoundTripTcp}}: *46 us*
* {{RoundTripServer}}: *49 us*
Note that the client/server classes used by {{RoundTripServer}} are also used by the TCP transport (configured in {{tcp.xml}}.
{{RoundTripServer}} is ~6% slower than {{RoundTripTcp}}, but that can be attributed to the additional work the former has to do (e.g. connection reaping etc). This is something we can focus on later.
The big difference are the 110 us for {{RoundTrip}}. The goal is to find out what causes this and eliminate it. Since {{RoundTrip}} and {{RoundTripServer}} use the same underlying client/server classes in JGroups, let's compare these 2.
Tasks:
* Remove all protocols other than TCP in the running stack: (e.g. {{probe.sh remove-protocol=MFC}}). I already did this and the diff was negligible, but let's run this again
* Try various bundlers (e.g. NoBundler)
* Reduce threads in the threadpools, possibly disable (regular and OOB) thread pools (replace with DirectExecutor)
* The default is 1 sender thread, but try with multiple threads
> RoundTrip: latency is high compared to RoundTripTcp/RoundTripServer
> -------------------------------------------------------------------
>
> Key: JGRP-2065
> URL: https://issues.jboss.org/browse/JGRP-2065
> Project: JGroups
> Issue Type: Task
> Reporter: Bela Ban
> Assignee: Bela Ban
> Fix For: 4.0
>
>
> {{RoundTrip}} is a simple test between 2 members measuring round-trip latency. The sender continually sends a message, the receiver receives it and sends the response, and the sender unblocks when the response is received. Then the sender sends the next message.
> The time for the request is then logged at the sender and a min/avg/max value is computed (probably changed to histograms later).
> {{RoundTrip}} uses a JGroups channel, configured with {{-props}}, e.g. {{-props ~/tcp.xml}}.
> {{RoundTripTcp}} does the same, but uses direct TCP sockets (no JGroups) for communication.
> {{RoundTripServer}} uses the client-server classes of JGroups for communication, but no channel is used.
> Round trip times (both processes on the same box, a Mac mini):
> * {{RoundTrip}} (with {{tcp.xml}} shipped with JGroups): *110 us*
> * {{RoundTripTcp}}: *46 us*
> * {{RoundTripServer}}: *49 us*
> Note that the client/server classes used by {{RoundTripServer}} are also used by the TCP transport (configured in {{tcp.xml}}.
> {{RoundTripServer}} is ~6% slower than {{RoundTripTcp}}, but that can be attributed to the additional work the former has to do (e.g. connection reaping etc). This is something we can focus on later.
> The big difference are the 110 us for {{RoundTrip}}. The goal is to find out what causes this and eliminate it. Since {{RoundTrip}} and {{RoundTripServer}} use the same underlying client/server classes in JGroups, let's compare these 2.
> Tasks:
> * Remove all protocols other than TCP in the running stack: (e.g. {{probe.sh remove-protocol=MFC}}). I already did this and the diff was negligible, but let's run this again
> * Try various bundlers (e.g. NoBundler)
> * Reduce threads in the threadpools, possibly disable (regular and OOB) thread pools (replace with DirectExecutor)
> * The default is 1 sender thread, but try with multiple threads
> * Measure perf between sending a message (in the bundler) and receiving it
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 11 months