[JBoss JIRA] (ISPN-3791) Silence "Received invalid rebalance confirmation from NodeX" exceptions
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-3791?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-3791:
-------------------------------
Fix Version/s: 9.4.1.Final
10.0.0.Alpha1
> Silence "Received invalid rebalance confirmation from NodeX" exceptions
> -----------------------------------------------------------------------
>
> Key: ISPN-3791
> URL: https://issues.jboss.org/browse/ISPN-3791
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 6.0.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Minor
> Fix For: 9.4.1.Final, 10.0.0.Alpha1
>
>
> When the coordinator shuts down, it tries to shut down each of its caches first. This triggers a rebalance for the rest of the members, but the rebalance usually finishes only after the coordinator's channel also shuts down.
> The nodes who finish their state transfer will then send a REBALANCE_CONFIRM command to the new coordinator, but the new coordinator doesn't know about that rebalance (it will start the rebalance process from scratch). This results in exceptions like this in the new coordinator's log:
> {noformat}
> 12:36:04,977 WARN [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread-2,ISPN-Node-1) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=MyCoolCache, type=REBALANCE_CONFIRM, sender=ISPN-Node-3-54019, joinInfo=null, topologyId=8, currentCH=null, pendingCH=null, throwable=null, viewId=4}: org.infinispan.commons.CacheException: Received invalid rebalance confirmation from ISPN-Node-3-54019 for cache MyCoolCache, we don't have a rebalance in progress
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:190) [infinispan-core-6.0.0.Final.jar:6.0.0.Final]
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:147) [infinispan-core-6.0.0.Final.jar:6.0.0.Final]
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124) [infinispan-core-6.0.0.Final.jar:6.0.0.Final]
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270) [infinispan-core-6.0.0.Final.jar:6.0.0.Final]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
> {noformat}
> A simple way to avoid these warnings would be to keep track of the coordinator that initiated a particular rebalance on each node, and only send the confirmation message to that coordinator. The same warnings seem to appear on the old coordinator, when it receives a confirmation after its ClusterTopologyManager started shutting down, so we may need another check there.
> A more ambitious approach would be to keep the old rebalance when the new coordinator takes over, and have another round in the cluster state recovery asking if any members have already sent REBALANCE_CONFIRMATION commands (after the new coordinator is ready to process those commands). This should eliminate the duplicate state transfer that happens now.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 5 months
[JBoss JIRA] (ISPN-3791) Silence "Received invalid rebalance confirmation from NodeX" exceptions
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-3791?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-3791:
-------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/6350
> Silence "Received invalid rebalance confirmation from NodeX" exceptions
> -----------------------------------------------------------------------
>
> Key: ISPN-3791
> URL: https://issues.jboss.org/browse/ISPN-3791
> Project: Infinispan
> Issue Type: Bug
> Components: State Transfer
> Affects Versions: 6.0.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Minor
>
> When the coordinator shuts down, it tries to shut down each of its caches first. This triggers a rebalance for the rest of the members, but the rebalance usually finishes only after the coordinator's channel also shuts down.
> The nodes who finish their state transfer will then send a REBALANCE_CONFIRM command to the new coordinator, but the new coordinator doesn't know about that rebalance (it will start the rebalance process from scratch). This results in exceptions like this in the new coordinator's log:
> {noformat}
> 12:36:04,977 WARN [org.infinispan.topology.CacheTopologyControlCommand] (remote-thread-2,ISPN-Node-1) ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=MyCoolCache, type=REBALANCE_CONFIRM, sender=ISPN-Node-3-54019, joinInfo=null, topologyId=8, currentCH=null, pendingCH=null, throwable=null, viewId=4}: org.infinispan.commons.CacheException: Received invalid rebalance confirmation from ISPN-Node-3-54019 for cache MyCoolCache, we don't have a rebalance in progress
> at org.infinispan.topology.ClusterTopologyManagerImpl.handleRebalanceCompleted(ClusterTopologyManagerImpl.java:190) [infinispan-core-6.0.0.Final.jar:6.0.0.Final]
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:147) [infinispan-core-6.0.0.Final.jar:6.0.0.Final]
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124) [infinispan-core-6.0.0.Final.jar:6.0.0.Final]
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher$4.run(CommandAwareRpcDispatcher.java:270) [infinispan-core-6.0.0.Final.jar:6.0.0.Final]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
> {noformat}
> A simple way to avoid these warnings would be to keep track of the coordinator that initiated a particular rebalance on each node, and only send the confirmation message to that coordinator. The same warnings seem to appear on the old coordinator, when it receives a confirmation after its ClusterTopologyManager started shutting down, so we may need another check there.
> A more ambitious approach would be to keep the old rebalance when the new coordinator takes over, and have another round in the cluster state recovery asking if any members have already sent REBALANCE_CONFIRMATION commands (after the new coordinator is ready to process those commands). This should eliminate the duplicate state transfer that happens now.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 5 months
[JBoss JIRA] (ISPN-9629) Coverage profile is not working
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9629?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant reassigned ISPN-9629:
-------------------------------------
Assignee: Diego Lovison
> Coverage profile is not working
> -------------------------------
>
> Key: ISPN-9629
> URL: https://issues.jboss.org/browse/ISPN-9629
> Project: Infinispan
> Issue Type: Bug
> Reporter: Diego Lovison
> Assignee: Diego Lovison
> Priority: Major
> Fix For: 9.4.1.Final
>
>
> When running the `coverage` profile. The build is failing because
> Run: mvn verify -Pcoverage -Dmaven.test.failure.ignore=true
> {noformat}
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (copy-generated-classes) on project infinispan-parent: Execution copy-generated-classes of goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run failed: Plugin org.apache.maven.plugins:maven-antrun-plugin:1.8 or one of its dependencies could not be resolved: Failure to find org.apache.ant:ant-nodeps:jar:1.10.1 in http://download.lab.eng.bos.redhat.com/brewroot/repos/jb-dg-7-rhel-7-buil... was cached in the local repository, resolution will not be reattempted until the update interval of mead-jdg7-plugins has elapsed or updates are forced -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please read the following articles:
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 5 months
[JBoss JIRA] (ISPN-9629) Coverage profile is not working
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9629?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9629:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 9.4.1.Final
Resolution: Done
> Coverage profile is not working
> -------------------------------
>
> Key: ISPN-9629
> URL: https://issues.jboss.org/browse/ISPN-9629
> Project: Infinispan
> Issue Type: Bug
> Reporter: Diego Lovison
> Priority: Major
> Fix For: 9.4.1.Final
>
>
> When running the `coverage` profile. The build is failing because
> Run: mvn verify -Pcoverage -Dmaven.test.failure.ignore=true
> {noformat}
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (copy-generated-classes) on project infinispan-parent: Execution copy-generated-classes of goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run failed: Plugin org.apache.maven.plugins:maven-antrun-plugin:1.8 or one of its dependencies could not be resolved: Failure to find org.apache.ant:ant-nodeps:jar:1.10.1 in http://download.lab.eng.bos.redhat.com/brewroot/repos/jb-dg-7-rhel-7-buil... was cached in the local repository, resolution will not be reattempted until the update interval of mead-jdg7-plugins has elapsed or updates are forced -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please read the following articles:
> {noformat}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 5 months
[JBoss JIRA] (ISPN-9626) RocksDB documentation is not compiling because of properties are configured wrong
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9626?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9626:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 9.4.1.Final
Assignee: Diego Lovison
Resolution: Done
> RocksDB documentation is not compiling because of properties are configured wrong
> ---------------------------------------------------------------------------------
>
> Key: ISPN-9626
> URL: https://issues.jboss.org/browse/ISPN-9626
> Project: Infinispan
> Issue Type: Bug
> Reporter: Diego Lovison
> Assignee: Diego Lovison
> Priority: Minor
> Fix For: 9.4.1.Final
>
>
> {code:java}
> Properties props = new Properties();
> props.addProperty("database.max_background_compactions", "2");
> props.addProperty("data.write_buffer_size", "512MB");
> {code}
> should be
> {code:java}
> Properties props = new Properties();
> props.put("database.max_background_compactions", "2");
> props.put("data.write_buffer_size", "512MB");
> {code}
> -----
> {code:xml}
> <property name="database.max_background_compactions" value="2"/>
> <property name="data.write_buffer_size" value="512MB"/>
> {code}
> should be
> {code:xml}
> <property name="database.max_background_compactions">2</property>
> <property name="data.write_buffer_size">512MB</property>
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 5 months
[JBoss JIRA] (ISPN-9616) Server does not include multimap
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9616?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9616:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 9.4.1.Final
Resolution: Done
> Server does not include multimap
> --------------------------------
>
> Key: ISPN-9616
> URL: https://issues.jboss.org/browse/ISPN-9616
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Reporter: Katia Aresti
> Assignee: Katia Aresti
> Priority: Major
> Fix For: 9.4.1.Final
>
>
> Using multimap remote
> INFO: ISPN004021: Infinispan version: 9.4.0.Final
> Oct 18, 2018 11:27:10 AM org.infinispan.client.hotrod.impl.protocol.Codec20 checkForErrorsInResponseStatus
> WARN: ISPN004005: Error received from the server: java.lang.NoClassDefFoundError: org/infinispan/multimap/impl/EmbeddedMultimapCache
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years, 5 months