[JBoss JIRA] (ISPN-2864) org.infinispan.client.hotrod.ReplTopologyChangeTest.testDropServer randomly fails on all environments
by Antonin Stefanutti (JIRA)
[ https://issues.jboss.org/browse/ISPN-2864?page=com.atlassian.jira.plugin.... ]
Antonin Stefanutti commented on ISPN-2864:
------------------------------------------
I'm reproducing that issue for remote query operation over Hot Rod Java client on:
* Windows 7 64 bits / Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)
* SunOS 5.10 i86pc i386 i86pc / Java HotSpot(TM) Server VM (build 10.0-b22, mixed mode)
With the following stacktrace:
{code}
2014-01-27 14:46:57,401 [pool-1-thread-1] ERROR org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory: ISPN004017: Could not fetch transport
org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: XXXXX/XXX.XXX.XXXX.XXX:6000
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:74)
at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:35)
at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:16)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:287)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:165)
at org.infinispan.client.hotrod.impl.operations.QueryOperation.getTransport(QueryOperation.java:38)
at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:45)
at org.infinispan.client.hotrod.impl.query.RemoteQuery.executeQuery(RemoteQuery.java:79)
at org.infinispan.client.hotrod.impl.query.RemoteQuery.list(RemoteQuery.java:69)
at monitoring.client.Main$1.run(Main.java:71)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:100)
at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:65)
... 19 more
2014-01-27 14:46:57,403 [pool-1-thread-1] WARN org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory: ISPN004022: Unable to invalidate transport for server: XXXXX/XXX.XXX.XXX.XXXX:6000
{code}
> org.infinispan.client.hotrod.ReplTopologyChangeTest.testDropServer randomly fails on all environments
> -----------------------------------------------------------------------------------------------------
>
> Key: ISPN-2864
> URL: https://issues.jboss.org/browse/ISPN-2864
> Project: Infinispan
> Issue Type: Bug
> Components: Remote Protocols
> Affects Versions: 5.2.2.Final, 6.0.0.Final, 6.0.1.Final
> Environment: Windows 2008 R2 64bit, Oracle JDK 7; Windows 2008 R2 64bit, Oracle JDK 6
> Solaris, Oracle JDK 6 and 7
> Reporter: Vitalii Chepeliuk
> Assignee: Galder Zamarreño
> Labels: testsuite_stability
> Fix For: 6.0.0.CR1
>
> Attachments: ReplTopologyChangeTest.log.zip
>
>
> localhost/127.0.0.1:15242 not found in [/127.0.0.1:15242, /127.0.0.1:15240, /127.0.0.1:15241] expected:<false> but was:<true>
> Stacktrace
> java.lang.AssertionError: localhost/127.0.0.1:15242 not found in [/127.0.0.1:15242, /127.0.0.1:15240, /127.0.0.1:15241] expected:<false> but was:<true>
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59)
> at org.testng.AssertJUnit.failNotEquals(AssertJUnit.java:364)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:80)
> at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:185)
> at org.infinispan.client.hotrod.ReplTopologyChangeTest.expectTopologyChange(ReplTopologyChangeTest.java:141)
> at org.infinispan.client.hotrod.ReplTopologyChangeTest.testDropServer(ReplTopologyChangeTest.java:123)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
> at org.testng.internal.Invoker.invokeMethod(Invoker.java:715)
> at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:907)
> at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1237)
> at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:127)
> at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:111)
> at org.testng.TestRunner.privateRun(TestRunner.java:767)
> at org.testng.TestRunner.run(TestRunner.java:617)
> at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
> at org.testng.SuiteRunner.access$000(SuiteRunner.java:37)
> at org.testng.SuiteRunner$SuiteWorker.run(SuiteRunner.java:368)
> at org.testng.internal.thread.ThreadUtil$2.call(ThreadUtil.java:64)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:662)
> Jenkins job could be found here.
> https://jenkins.mw.lab.eng.bos.redhat.com/hudson/view/JDG/view/FUNC/job/e...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-3938) AdvancedAsyncCacheLoader.process() concurrency issues
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-3938?page=com.atlassian.jira.plugin.... ]
William Burns commented on ISPN-3938:
-------------------------------------
Whether process method returns after it is iterated over the keys is defined by the cache loader implementation though, so an implementation can do whatever it wants there. If you look at the references to allKeys which threw the ConcurrentModificationException it is only possible if the process method returned early.
{code}
Set<Object> allKeys = new HashSet<Object>(batchSize);
Set<Object> batch = new HashSet<Object>();
loadAllKeys(state.get(), allKeys, keyFilter, executor); <--- reference
for (Iterator it = allKeys.iterator(); it.hasNext(); ) { <--- reference in for loop (doesn't pass reference to any other method invocations)
batch.add(it.next());
if (batch.size() == batchSize) {
final Set<Object> toHandle = batch;
batch = new HashSet<Object>(batchSize);
submitProcessTask(cacheLoaderTask, eacs, taskContext, toHandle);
}
}
-- No more references to the HashSet past this.
{code}
The HashSet is only used in the loadAllKeys method and the iterator. The iterator is guaranteed to not be used concurrently with the code shown. The loadAllKeys however can with the process method. Or is there somewhere else I am not seeing?
> AdvancedAsyncCacheLoader.process() concurrency issues
> -----------------------------------------------------
>
> Key: ISPN-3938
> URL: https://issues.jboss.org/browse/ISPN-3938
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Loaders and Stores
> Affects Versions: 6.0.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 7.0.0.Final
>
>
> {{AdvancedAsyncCacheLoader.process()}} calls {{advancedLoader().process()}} to collect all the keys in the store, but the HashSet used to collect the keys it not thread-safe. This can cause problems, e.g. during state transfer:
> {noformat}
> WARN cheTopologyControlCommand | ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=sessions, type=CH_UPDATE, sender=alfie-lt-46127, joinInfo=null, topologyId=3, currentCH=DefaultConsistentHash{numSegments=60, numOwners=1, members=[alfie-lt-46127]}, pendingCH=null, throwable=null, viewId=1}java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$KeyIterator.next(HashMap.java:960)
> at org.infinispan.persistence.async.AdvancedAsyncCacheLoader.process(AdvancedAsyncCacheLoader.java:80)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:414)
> at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:910)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:393)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:100)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleConsistentHashUpdate(LocalTopologyManagerImpl.java:191)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:152)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)
> at org.infinispan.topology.ClusterTopologyManagerImpl$3.run(ClusterTopologyManagerImpl.java:606)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-3942) HotRod client keep trying recover a connection to a cluster member after shutdown
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3942?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3942:
--------------------------------
Assignee: Galder Zamarreño (was: Mircea Markus)
> HotRod client keep trying recover a connection to a cluster member after shutdown
> ---------------------------------------------------------------------------------
>
> Key: ISPN-3942
> URL: https://issues.jboss.org/browse/ISPN-3942
> Project: Infinispan
> Issue Type: Bug
> Components: Remote Protocols
> Affects Versions: 6.0.1.Final, 7.0.0.Alpha1
> Environment: Infinispan server with HotRod client in server-client mode and replicated caches.
> Reporter: Wolf-Dieter Fink
> Assignee: Galder Zamarreño
> Labels: clustered, hotrod, hotrod-java-client
> Attachments: hotrod-client.log
>
>
> If a HotRod client is connected to a server cluster and one of the servers do a correct shutdown, the client keep try to reconnect the server after the cluster view has changed.
> There is only a replicated cache configured.
> From the server-logfile the new cluster view is established
> 16:00:22,290 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-3,shared=udp) ISPN000094: Received new cluster view: [node1/clustered|2] (1) [node1/clustered]
> There is no failure from the application perspective, but there there is always a retry if the RoundRobin return the unavailable server.
> This might end in a performance or failure issue if there is a larger cluster and a part going out of work for some reason.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-3938) AdvancedAsyncCacheLoader.process() concurrency issues
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-3938?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-3938:
------------------------------------
[~william.burns] I think the iteration does wait for all the keys to be processed, because {{AdvancedCacheLoader.process()}} is guaranteed to return only when all the entries have been processed. Implementations seem to use an {{ExecutorAllCompletionService}} to provide that guarantee.
> AdvancedAsyncCacheLoader.process() concurrency issues
> -----------------------------------------------------
>
> Key: ISPN-3938
> URL: https://issues.jboss.org/browse/ISPN-3938
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Loaders and Stores
> Affects Versions: 6.0.0.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Fix For: 7.0.0.Final
>
>
> {{AdvancedAsyncCacheLoader.process()}} calls {{advancedLoader().process()}} to collect all the keys in the store, but the HashSet used to collect the keys it not thread-safe. This can cause problems, e.g. during state transfer:
> {noformat}
> WARN cheTopologyControlCommand | ISPN000071: Caught exception when handling command CacheTopologyControlCommand{cache=sessions, type=CH_UPDATE, sender=alfie-lt-46127, joinInfo=null, topologyId=3, currentCH=DefaultConsistentHash{numSegments=60, numOwners=1, members=[alfie-lt-46127]}, pendingCH=null, throwable=null, viewId=1}java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:926)
> at java.util.HashMap$KeyIterator.next(HashMap.java:960)
> at org.infinispan.persistence.async.AdvancedAsyncCacheLoader.process(AdvancedAsyncCacheLoader.java:80)
> at org.infinispan.persistence.manager.PersistenceManagerImpl.processOnAllStores(PersistenceManagerImpl.java:414)
> at org.infinispan.statetransfer.StateConsumerImpl.invalidateSegments(StateConsumerImpl.java:910)
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:393)
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:178)
> at org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:38)
> at org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:100)
> at org.infinispan.topology.LocalTopologyManagerImpl.handleConsistentHashUpdate(LocalTopologyManagerImpl.java:191)
> at org.infinispan.topology.CacheTopologyControlCommand.doPerform(CacheTopologyControlCommand.java:152)
> at org.infinispan.topology.CacheTopologyControlCommand.perform(CacheTopologyControlCommand.java:124)
> at org.infinispan.topology.ClusterTopologyManagerImpl$3.run(ClusterTopologyManagerImpl.java:606)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-3942) HotRod client keep trying recover a connection to a cluster member after shutdown
by Wolf-Dieter Fink (JIRA)
[ https://issues.jboss.org/browse/ISPN-3942?page=com.atlassian.jira.plugin.... ]
Wolf-Dieter Fink updated ISPN-3942:
-----------------------------------
Attachment: hotrod-client.log
Client side log with ISPN 7.0 (upstream)
> HotRod client keep trying recover a connection to a cluster member after shutdown
> ---------------------------------------------------------------------------------
>
> Key: ISPN-3942
> URL: https://issues.jboss.org/browse/ISPN-3942
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 6.0.1.Final, 7.0.0.Alpha1
> Environment: Infinispan server with HotRod client in server-client mode and replicated caches.
> Reporter: Wolf-Dieter Fink
> Assignee: Mircea Markus
> Labels: clustered, hotrod, hotrod-java-client
> Attachments: hotrod-client.log
>
>
> If a HotRod client is connected to a server cluster and one of the servers do a correct shutdown, the client keep try to reconnect the server after the cluster view has changed.
> There is only a replicated cache configured.
> From the server-logfile the new cluster view is established
> 16:00:22,290 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-3,shared=udp) ISPN000094: Received new cluster view: [node1/clustered|2] (1) [node1/clustered]
> There is no failure from the application perspective, but there there is always a retry if the RoundRobin return the unavailable server.
> This might end in a performance or failure issue if there is a larger cluster and a part going out of work for some reason.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months
[JBoss JIRA] (ISPN-3942) HotRod client keep trying recover a connection to a cluster member after shutdown
by Wolf-Dieter Fink (JIRA)
[ https://issues.jboss.org/browse/ISPN-3942?page=com.atlassian.jira.plugin.... ]
Wolf-Dieter Fink updated ISPN-3942:
-----------------------------------
Component/s: Remote Protocols
> HotRod client keep trying recover a connection to a cluster member after shutdown
> ---------------------------------------------------------------------------------
>
> Key: ISPN-3942
> URL: https://issues.jboss.org/browse/ISPN-3942
> Project: Infinispan
> Issue Type: Bug
> Components: Remote Protocols
> Affects Versions: 6.0.1.Final, 7.0.0.Alpha1
> Environment: Infinispan server with HotRod client in server-client mode and replicated caches.
> Reporter: Wolf-Dieter Fink
> Assignee: Mircea Markus
> Labels: clustered, hotrod, hotrod-java-client
> Attachments: hotrod-client.log
>
>
> If a HotRod client is connected to a server cluster and one of the servers do a correct shutdown, the client keep try to reconnect the server after the cluster view has changed.
> There is only a replicated cache configured.
> From the server-logfile the new cluster view is established
> 16:00:22,290 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (Incoming-3,shared=udp) ISPN000094: Received new cluster view: [node1/clustered|2] (1) [node1/clustered]
> There is no failure from the application perspective, but there there is always a retry if the RoundRobin return the unavailable server.
> This might end in a performance or failure issue if there is a larger cluster and a part going out of work for some reason.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 11 months