[JBoss JIRA] (ISPN-2836) org.jgroups.TimeoutException after invoking MapCombineCommand in Map/Reduce task with 2 nodes
by Alan Field (JIRA)
[ https://issues.jboss.org/browse/ISPN-2836?page=com.atlassian.jira.plugin.... ]
Alan Field commented on ISPN-2836:
----------------------------------
[~pruivo] OK, so I have a couple of questions:
1) Can you send me the updated executeMapReduceTask() code?
2) Does the cache configuration need to be changed to avoid the TimeoutException with larger amounts of data in the cache?
> org.jgroups.TimeoutException after invoking MapCombineCommand in Map/Reduce task with 2 nodes
> ---------------------------------------------------------------------------------------------
>
> Key: ISPN-2836
> URL: https://issues.jboss.org/browse/ISPN-2836
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.2.1.Final
> Reporter: Alan Field
> Assignee: Pedro Ruivo
> Priority: Blocker
> Labels: onboard
> Fix For: 5.3.0.Final
>
> Attachments: afield-tcp-521-final.txt, benchmark-mapreduce-multifilesize.xml, dist-udp-no-tx.xml, jgroups-udp.xml, udp-edg-perf01.txt, udp-edg-perf02.txt
>
>
> Using RadarGun and two nodes to execute the example WordCount Map/Reduce job against a cache with ~550 keys with a value size of 1MB is producing a thread deadlock. The cache is distributed with transactions disabled.
> TCP transport deadlocks without throwing an exception. Disabling the send queue and setting UNICAST2.conn_expiry_timeout=0 prevents the deadlock, but the job does not complete. The nodes send "are-you-alive" messages back and forth, and I have seen the following exception:
> {noformat}
> 11:44:29,970 ERROR [org.jgroups.protocols.TCP] (OOB-98,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (76 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:352)
> at org.radargun.cachewrappers.InfinispanMapReduceWrapper.executeMapReduceTask(InfinispanMapReduceWrapper.java:98)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:74)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:832)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhaseWithLocalReduction(MapReduceTask.java:477)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:350)
> ... 9 more
> Caused by: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at org.infinispan.util.Util.rewrapAsCacheException(Util.java:541)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
> 11:44:29,978 ERROR [org.jgroups.protocols.TCP] (Timer-3,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (60 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:175)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:254)
> at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:80)
> at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:288)
> ... 5 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:390)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301)
> 11:44:29,979 ERROR [org.jgroups.protocols.TCP] (Timer-4,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (63 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)
> ... 11 more
> {noformat}
> With UDP transport, both threads are deadlocked. I will attach thread dumps from runs using TCP and UDP transport.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 10 months
[JBoss JIRA] (ISPN-3163) Replacing entry via HotRod which was initially stored via Memcached does not change CAS
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-3163?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-3163:
----------------------------------------
The best is probably to have type converter generate a new version and assign to the metadata in the command, if no version given by the client. For this to work, a way to share the version generator is needed by all parties (type converter interceptor and server endpoints). That way, they can all share the same version generator and guarantees that at the local level, increasing numbers are provided (different nodes will generate different versions already...).
> Replacing entry via HotRod which was initially stored via Memcached does not change CAS
> ---------------------------------------------------------------------------------------
>
> Key: ISPN-3163
> URL: https://issues.jboss.org/browse/ISPN-3163
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.3.0.CR1
> Reporter: Martin Gencur
> Assignee: Galder Zamarreño
> Fix For: 5.3.0.Final
>
>
> Users might expect that CAS (check-and-set) operation will work even in compatibility mode which is currently not true in the following scenario:
> 1) store a key/value via Memcached
> 2) change the value via HotRod or Embedded
> 3) use Memcached's CAS operation
> In step #3 the memcached client will update the value even though the value was changed by another client in the meantime. The memcached client was supposed to change it only if it had not been changed in the meantime.
> The following test snippet shows the problem:
> {code:java}
> public void testMemcachedPutHotRodEmbbeddedReplaceMemcachedCASTest() throws Exception {
> final String key1 = "5";
> // 1. Put with Memcached
> Future<Boolean> f = cacheFactory.getMemcachedClient().set(key1, 0, "v1");
> assertTrue(f.get(60, TimeUnit.SECONDS));
> CASValue oldValue = cacheFactory.getMemcachedClient().gets(key1);
> // 2. Replace with Hot Rod
> VersionedValue versioned = cacheFactory.getHotRodCache().getVersioned(key1);
> assertTrue(cacheFactory.getHotRodCache().replaceWithVersion(key1, "v2", versioned.getVersion()));
> // 3. Replace with Embedded
> assertTrue(cacheFactory.getEmbeddedCache().replace(key1, "v2", "v3"));
> // 4. Get with Memcached and verify value/CAS
> CASValue newValue = cacheFactory.getMemcachedClient().gets(key1);
> assertEquals("v3", newValue.getValue());
> assertTrue("The version (CAS) should have changed", oldValue.getCas() != newValue.getCas());
> //<---- fails here
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 10 months
[JBoss JIRA] (ISPN-3163) Replacing entry via HotRod which was initially stored via Memcached does not change CAS
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-3163?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-3163:
----------------------------------------
In fact, there's already something for that: org.infinispan.container.versioning.VersionGenerator...
> Replacing entry via HotRod which was initially stored via Memcached does not change CAS
> ---------------------------------------------------------------------------------------
>
> Key: ISPN-3163
> URL: https://issues.jboss.org/browse/ISPN-3163
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 5.3.0.CR1
> Reporter: Martin Gencur
> Assignee: Galder Zamarreño
> Fix For: 5.3.0.Final
>
>
> Users might expect that CAS (check-and-set) operation will work even in compatibility mode which is currently not true in the following scenario:
> 1) store a key/value via Memcached
> 2) change the value via HotRod or Embedded
> 3) use Memcached's CAS operation
> In step #3 the memcached client will update the value even though the value was changed by another client in the meantime. The memcached client was supposed to change it only if it had not been changed in the meantime.
> The following test snippet shows the problem:
> {code:java}
> public void testMemcachedPutHotRodEmbbeddedReplaceMemcachedCASTest() throws Exception {
> final String key1 = "5";
> // 1. Put with Memcached
> Future<Boolean> f = cacheFactory.getMemcachedClient().set(key1, 0, "v1");
> assertTrue(f.get(60, TimeUnit.SECONDS));
> CASValue oldValue = cacheFactory.getMemcachedClient().gets(key1);
> // 2. Replace with Hot Rod
> VersionedValue versioned = cacheFactory.getHotRodCache().getVersioned(key1);
> assertTrue(cacheFactory.getHotRodCache().replaceWithVersion(key1, "v2", versioned.getVersion()));
> // 3. Replace with Embedded
> assertTrue(cacheFactory.getEmbeddedCache().replace(key1, "v2", "v3"));
> // 4. Get with Memcached and verify value/CAS
> CASValue newValue = cacheFactory.getMemcachedClient().gets(key1);
> assertEquals("v3", newValue.getValue());
> assertTrue("The version (CAS) should have changed", oldValue.getCas() != newValue.getCas());
> //<---- fails here
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 10 months
[JBoss JIRA] (ISPN-3182) Compatibility mode does not work correctly in ISPN server
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-3182?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-3182:
----------------------------------------
Or the alternative getCache[Array[Byte], Array[Byte]](cacheName) call...
> Compatibility mode does not work correctly in ISPN server
> ---------------------------------------------------------
>
> Key: ISPN-3182
> URL: https://issues.jboss.org/browse/ISPN-3182
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Reporter: Martin Gencur
> Assignee: Tristan Tarrant
> Priority: Critical
> Fix For: 5.3.0.Final
>
>
> ISPN version: f6d1761 - Galder Zamarreño, ISPN-3062 Push JCache TCK output to screen
> Using <compatibility enabled="true"> , storing an antry via HotRod, and reading it again via HotRod fails -> null is returned.
> If the compatibility mode is disabled, the HotRod's get() operation returns the correct value.
> The following code snippet illustrates the bug:
> {code:java}
> @Test
> public void testHotRodPutRestGetTest() throws Exception {
> final String key = "1";
> // 1. Put with Hot Rod
> RemoteCache<String, byte[]> remote = getHotRodCache();
> assertEquals(null, remote.withFlags(Flag.FORCE_RETURN_VALUE).put(key, "v1".getBytes()));
> assertArrayEquals("v1".getBytes(), remote.get(key));
> //^^^^ --------- fails here
> // 2. Get with REST
> HttpMethod get = new GetMethod(getRestUrl() + "/" + key);
> getRestClient().executeMethod(get);
> assertEquals(HttpServletResponse.SC_OK, get.getStatusCode());
> assertEquals("v1".getBytes(), get.getResponseBody());
> }
> {code}
> The tests along with the example configuration file can be found at https://github.com/mgencur/infinispan-server/tree/ISPN-3176/compatibility...
> The relevant part of TRACE log is here:
> {code}
> 14:54:57,066 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (HotRodServerWorker-3) Successfully acquired lock 1!
> 14:54:57,066 TRACE [org.infinispan.interceptors.EntryWrappingInterceptor] (HotRodServerWorker-3) Wrapping entry '1'? true
> 14:54:57,066 TRACE [org.infinispan.container.EntryFactoryImpl] (HotRodServerWorker-3) Exists in context? null
> 14:54:57,067 TRACE [org.infinispan.container.EntryFactoryImpl] (HotRodServerWorker-3) Retrieved from container null
> 14:54:57,067 TRACE [org.infinispan.container.EntryFactoryImpl] (HotRodServerWorker-3) Creating new entry.
> 14:54:57,069 TRACE [org.infinispan.interceptors.CallInterceptor] (HotRodServerWorker-3) Executing command: PutKeyValueCommand{key=1, value=v1, flags=[OPERATION_HOTROD], putIfAbsent=false, metadata=EmbeddedMetadata{version=ServerEntryVersion(1)}, successful=true}.
> 14:54:57,070 TRACE [org.infinispan.interceptors.EntryWrappingInterceptor] (HotRodServerWorker-3) About to commit entry ReadCommittedEntry(74161820){key=1, value=v1, oldValue=null, isCreated=true, isChanged=true, isRemoved=false, isValid=true}
> 14:54:57,071 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (HotRodServerWorker-3) Updating entry (key=1 removed=false valid=true changed=true created=true loaded=false value=v1]
> 14:54:57,071 TRACE [org.infinispan.interceptors.EntryWrappingInterceptor] (HotRodServerWorker-3) The return value is null
> 14:54:57,072 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (HotRodServerWorker-3) Attempting to unlock 1
> 14:54:57,073 TRACE [org.infinispan.util.concurrent.locks.containers.ReentrantPerEntryLockContainer] (HotRodServerWorker-3) Unlocking lock instance for key 1
> 14:54:57,074 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) Write response ResponseWithPrevious{version=12, messageId=3, operation=PutResponse, status=Success, previous=null}
> 14:54:57,074 TRACE [org.infinispan.server.hotrod.HotRodEncoder] (HotRodServerWorker-3) Encode msg ResponseWithPrevious{version=12, messageId=3, operation=PutResponse, status=Success, previous=null}
> 14:54:57,075 TRACE [org.infinispan.server.hotrod.Encoders$Encoder12$] (HotRodServerWorker-3) Write topology response header with no change
> 14:54:57,075 TRACE [org.infinispan.server.hotrod.HotRodEncoder] (HotRodServerWorker-3) Write buffer contents A10302000000 to channel [id: 0x6ef9bcb3, /127.0.0.1:55677 => /127.0.0.1:11222]
> 14:54:57,078 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) Decode using instance @265b1dbd
> 14:54:57,078 TRACE [org.infinispan.server.hotrod.Decoder10$] (HotRodServerWorker-3) Operation code: 3 has been matched to GetRequest
> 14:54:57,079 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) Decoded header HotRodHeader{op=GetRequest, version=12, messageId=4, cacheName=, flag=0, clientIntelligence=3, topologyId=-1}
> 14:54:57,079 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) Decode using instance @265b1dbd
> 14:54:57,080 TRACE [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerWorker-3) Invoked with command GetKeyValueCommand {key=[B0x033e0131..(4), flags=null} and InvocationContext [org.infinispan.context.SingleKeyNonTxInvocationContext@721c2e8a]
> 14:54:57,080 TRACE [org.infinispan.container.EntryFactoryImpl] (HotRodServerWorker-3) Exists in context? null
> 14:54:57,080 TRACE [org.infinispan.container.EntryFactoryImpl] (HotRodServerWorker-3) Retrieved from container null
> 14:54:57,080 TRACE [org.infinispan.interceptors.CallInterceptor] (HotRodServerWorker-3) Executing command: GetKeyValueCommand {key=[B0x033e0131..(4), flags=null}.
> 14:54:57,081 TRACE [org.infinispan.commands.read.GetKeyValueCommand] (HotRodServerWorker-3) Entry not found
> 14:54:57,081 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) Write response GetResponse{version=12, messageId=4, operation=GetResponse, status=KeyDoesNotExist, data=null}
> 14:54:57,081 TRACE [org.infinispan.server.hotrod.HotRodEncoder] (HotRodServerWorker-3) Encode msg GetResponse{version=12, messageId=4, operation=GetResponse, status=KeyDoesNotExist, data=null}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 10 months
[JBoss JIRA] (ISPN-3182) Compatibility mode does not work correctly in ISPN server
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-3182?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-3182:
----------------------------------------
Or most likely, the problem might be that calling [1] does not have the same effect of caching the given flags as it does in standalone...
{code}[1] cacheManager.getCache[Array[Byte], Array[Byte]]().getAdvancedCache.withFlags(Flag.OPERATION_HOTROD){code}
> Compatibility mode does not work correctly in ISPN server
> ---------------------------------------------------------
>
> Key: ISPN-3182
> URL: https://issues.jboss.org/browse/ISPN-3182
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Reporter: Martin Gencur
> Assignee: Tristan Tarrant
> Priority: Critical
> Fix For: 5.3.0.Final
>
>
> ISPN version: f6d1761 - Galder Zamarreño, ISPN-3062 Push JCache TCK output to screen
> Using <compatibility enabled="true"> , storing an antry via HotRod, and reading it again via HotRod fails -> null is returned.
> If the compatibility mode is disabled, the HotRod's get() operation returns the correct value.
> The following code snippet illustrates the bug:
> {code:java}
> @Test
> public void testHotRodPutRestGetTest() throws Exception {
> final String key = "1";
> // 1. Put with Hot Rod
> RemoteCache<String, byte[]> remote = getHotRodCache();
> assertEquals(null, remote.withFlags(Flag.FORCE_RETURN_VALUE).put(key, "v1".getBytes()));
> assertArrayEquals("v1".getBytes(), remote.get(key));
> //^^^^ --------- fails here
> // 2. Get with REST
> HttpMethod get = new GetMethod(getRestUrl() + "/" + key);
> getRestClient().executeMethod(get);
> assertEquals(HttpServletResponse.SC_OK, get.getStatusCode());
> assertEquals("v1".getBytes(), get.getResponseBody());
> }
> {code}
> The tests along with the example configuration file can be found at https://github.com/mgencur/infinispan-server/tree/ISPN-3176/compatibility...
> The relevant part of TRACE log is here:
> {code}
> 14:54:57,066 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (HotRodServerWorker-3) Successfully acquired lock 1!
> 14:54:57,066 TRACE [org.infinispan.interceptors.EntryWrappingInterceptor] (HotRodServerWorker-3) Wrapping entry '1'? true
> 14:54:57,066 TRACE [org.infinispan.container.EntryFactoryImpl] (HotRodServerWorker-3) Exists in context? null
> 14:54:57,067 TRACE [org.infinispan.container.EntryFactoryImpl] (HotRodServerWorker-3) Retrieved from container null
> 14:54:57,067 TRACE [org.infinispan.container.EntryFactoryImpl] (HotRodServerWorker-3) Creating new entry.
> 14:54:57,069 TRACE [org.infinispan.interceptors.CallInterceptor] (HotRodServerWorker-3) Executing command: PutKeyValueCommand{key=1, value=v1, flags=[OPERATION_HOTROD], putIfAbsent=false, metadata=EmbeddedMetadata{version=ServerEntryVersion(1)}, successful=true}.
> 14:54:57,070 TRACE [org.infinispan.interceptors.EntryWrappingInterceptor] (HotRodServerWorker-3) About to commit entry ReadCommittedEntry(74161820){key=1, value=v1, oldValue=null, isCreated=true, isChanged=true, isRemoved=false, isValid=true}
> 14:54:57,071 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (HotRodServerWorker-3) Updating entry (key=1 removed=false valid=true changed=true created=true loaded=false value=v1]
> 14:54:57,071 TRACE [org.infinispan.interceptors.EntryWrappingInterceptor] (HotRodServerWorker-3) The return value is null
> 14:54:57,072 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (HotRodServerWorker-3) Attempting to unlock 1
> 14:54:57,073 TRACE [org.infinispan.util.concurrent.locks.containers.ReentrantPerEntryLockContainer] (HotRodServerWorker-3) Unlocking lock instance for key 1
> 14:54:57,074 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) Write response ResponseWithPrevious{version=12, messageId=3, operation=PutResponse, status=Success, previous=null}
> 14:54:57,074 TRACE [org.infinispan.server.hotrod.HotRodEncoder] (HotRodServerWorker-3) Encode msg ResponseWithPrevious{version=12, messageId=3, operation=PutResponse, status=Success, previous=null}
> 14:54:57,075 TRACE [org.infinispan.server.hotrod.Encoders$Encoder12$] (HotRodServerWorker-3) Write topology response header with no change
> 14:54:57,075 TRACE [org.infinispan.server.hotrod.HotRodEncoder] (HotRodServerWorker-3) Write buffer contents A10302000000 to channel [id: 0x6ef9bcb3, /127.0.0.1:55677 => /127.0.0.1:11222]
> 14:54:57,078 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) Decode using instance @265b1dbd
> 14:54:57,078 TRACE [org.infinispan.server.hotrod.Decoder10$] (HotRodServerWorker-3) Operation code: 3 has been matched to GetRequest
> 14:54:57,079 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) Decoded header HotRodHeader{op=GetRequest, version=12, messageId=4, cacheName=, flag=0, clientIntelligence=3, topologyId=-1}
> 14:54:57,079 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) Decode using instance @265b1dbd
> 14:54:57,080 TRACE [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerWorker-3) Invoked with command GetKeyValueCommand {key=[B0x033e0131..(4), flags=null} and InvocationContext [org.infinispan.context.SingleKeyNonTxInvocationContext@721c2e8a]
> 14:54:57,080 TRACE [org.infinispan.container.EntryFactoryImpl] (HotRodServerWorker-3) Exists in context? null
> 14:54:57,080 TRACE [org.infinispan.container.EntryFactoryImpl] (HotRodServerWorker-3) Retrieved from container null
> 14:54:57,080 TRACE [org.infinispan.interceptors.CallInterceptor] (HotRodServerWorker-3) Executing command: GetKeyValueCommand {key=[B0x033e0131..(4), flags=null}.
> 14:54:57,081 TRACE [org.infinispan.commands.read.GetKeyValueCommand] (HotRodServerWorker-3) Entry not found
> 14:54:57,081 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) Write response GetResponse{version=12, messageId=4, operation=GetResponse, status=KeyDoesNotExist, data=null}
> 14:54:57,081 TRACE [org.infinispan.server.hotrod.HotRodEncoder] (HotRodServerWorker-3) Encode msg GetResponse{version=12, messageId=4, operation=GetResponse, status=KeyDoesNotExist, data=null}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 10 months
[JBoss JIRA] (ISPN-3182) Compatibility mode does not work correctly in ISPN server
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-3182?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-3182:
----------------------------------------
That get command should be coming with OPERATION_HOTROD flag. When that flag is passed, it will use the Hot Rod type converter to box the key, so transform the byte[] to the String key with unmarshalling. Most likely this is happening due to slight differences in how the cache retrieval works in standalone vs within AS7/EAP... I'd debug HotRodServer.getCacheInstance() method...
> Compatibility mode does not work correctly in ISPN server
> ---------------------------------------------------------
>
> Key: ISPN-3182
> URL: https://issues.jboss.org/browse/ISPN-3182
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Reporter: Martin Gencur
> Assignee: Tristan Tarrant
> Priority: Critical
> Fix For: 5.3.0.Final
>
>
> ISPN version: f6d1761 - Galder Zamarreño, ISPN-3062 Push JCache TCK output to screen
> Using <compatibility enabled="true"> , storing an antry via HotRod, and reading it again via HotRod fails -> null is returned.
> If the compatibility mode is disabled, the HotRod's get() operation returns the correct value.
> The following code snippet illustrates the bug:
> {code:java}
> @Test
> public void testHotRodPutRestGetTest() throws Exception {
> final String key = "1";
> // 1. Put with Hot Rod
> RemoteCache<String, byte[]> remote = getHotRodCache();
> assertEquals(null, remote.withFlags(Flag.FORCE_RETURN_VALUE).put(key, "v1".getBytes()));
> assertArrayEquals("v1".getBytes(), remote.get(key));
> //^^^^ --------- fails here
> // 2. Get with REST
> HttpMethod get = new GetMethod(getRestUrl() + "/" + key);
> getRestClient().executeMethod(get);
> assertEquals(HttpServletResponse.SC_OK, get.getStatusCode());
> assertEquals("v1".getBytes(), get.getResponseBody());
> }
> {code}
> The tests along with the example configuration file can be found at https://github.com/mgencur/infinispan-server/tree/ISPN-3176/compatibility...
> The relevant part of TRACE log is here:
> {code}
> 14:54:57,066 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (HotRodServerWorker-3) Successfully acquired lock 1!
> 14:54:57,066 TRACE [org.infinispan.interceptors.EntryWrappingInterceptor] (HotRodServerWorker-3) Wrapping entry '1'? true
> 14:54:57,066 TRACE [org.infinispan.container.EntryFactoryImpl] (HotRodServerWorker-3) Exists in context? null
> 14:54:57,067 TRACE [org.infinispan.container.EntryFactoryImpl] (HotRodServerWorker-3) Retrieved from container null
> 14:54:57,067 TRACE [org.infinispan.container.EntryFactoryImpl] (HotRodServerWorker-3) Creating new entry.
> 14:54:57,069 TRACE [org.infinispan.interceptors.CallInterceptor] (HotRodServerWorker-3) Executing command: PutKeyValueCommand{key=1, value=v1, flags=[OPERATION_HOTROD], putIfAbsent=false, metadata=EmbeddedMetadata{version=ServerEntryVersion(1)}, successful=true}.
> 14:54:57,070 TRACE [org.infinispan.interceptors.EntryWrappingInterceptor] (HotRodServerWorker-3) About to commit entry ReadCommittedEntry(74161820){key=1, value=v1, oldValue=null, isCreated=true, isChanged=true, isRemoved=false, isValid=true}
> 14:54:57,071 TRACE [org.infinispan.container.entries.ReadCommittedEntry] (HotRodServerWorker-3) Updating entry (key=1 removed=false valid=true changed=true created=true loaded=false value=v1]
> 14:54:57,071 TRACE [org.infinispan.interceptors.EntryWrappingInterceptor] (HotRodServerWorker-3) The return value is null
> 14:54:57,072 TRACE [org.infinispan.util.concurrent.locks.LockManagerImpl] (HotRodServerWorker-3) Attempting to unlock 1
> 14:54:57,073 TRACE [org.infinispan.util.concurrent.locks.containers.ReentrantPerEntryLockContainer] (HotRodServerWorker-3) Unlocking lock instance for key 1
> 14:54:57,074 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) Write response ResponseWithPrevious{version=12, messageId=3, operation=PutResponse, status=Success, previous=null}
> 14:54:57,074 TRACE [org.infinispan.server.hotrod.HotRodEncoder] (HotRodServerWorker-3) Encode msg ResponseWithPrevious{version=12, messageId=3, operation=PutResponse, status=Success, previous=null}
> 14:54:57,075 TRACE [org.infinispan.server.hotrod.Encoders$Encoder12$] (HotRodServerWorker-3) Write topology response header with no change
> 14:54:57,075 TRACE [org.infinispan.server.hotrod.HotRodEncoder] (HotRodServerWorker-3) Write buffer contents A10302000000 to channel [id: 0x6ef9bcb3, /127.0.0.1:55677 => /127.0.0.1:11222]
> 14:54:57,078 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) Decode using instance @265b1dbd
> 14:54:57,078 TRACE [org.infinispan.server.hotrod.Decoder10$] (HotRodServerWorker-3) Operation code: 3 has been matched to GetRequest
> 14:54:57,079 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) Decoded header HotRodHeader{op=GetRequest, version=12, messageId=4, cacheName=, flag=0, clientIntelligence=3, topologyId=-1}
> 14:54:57,079 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) Decode using instance @265b1dbd
> 14:54:57,080 TRACE [org.infinispan.interceptors.InvocationContextInterceptor] (HotRodServerWorker-3) Invoked with command GetKeyValueCommand {key=[B0x033e0131..(4), flags=null} and InvocationContext [org.infinispan.context.SingleKeyNonTxInvocationContext@721c2e8a]
> 14:54:57,080 TRACE [org.infinispan.container.EntryFactoryImpl] (HotRodServerWorker-3) Exists in context? null
> 14:54:57,080 TRACE [org.infinispan.container.EntryFactoryImpl] (HotRodServerWorker-3) Retrieved from container null
> 14:54:57,080 TRACE [org.infinispan.interceptors.CallInterceptor] (HotRodServerWorker-3) Executing command: GetKeyValueCommand {key=[B0x033e0131..(4), flags=null}.
> 14:54:57,081 TRACE [org.infinispan.commands.read.GetKeyValueCommand] (HotRodServerWorker-3) Entry not found
> 14:54:57,081 TRACE [org.infinispan.server.hotrod.HotRodDecoder] (HotRodServerWorker-3) Write response GetResponse{version=12, messageId=4, operation=GetResponse, status=KeyDoesNotExist, data=null}
> 14:54:57,081 TRACE [org.infinispan.server.hotrod.HotRodEncoder] (HotRodServerWorker-3) Encode msg GetResponse{version=12, messageId=4, operation=GetResponse, status=KeyDoesNotExist, data=null}
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 10 months
[JBoss JIRA] (ISPN-2836) org.jgroups.TimeoutException after invoking MapCombineCommand in Map/Reduce task with 2 nodes
by Pedro Ruivo (JIRA)
[ https://issues.jboss.org/browse/ISPN-2836?page=com.atlassian.jira.plugin.... ]
Pedro Ruivo commented on ISPN-2836:
-----------------------------------
Found it! [~afield] the problem is the fact the exception is not being caught. In that case, the slaves will never send any ack back to the master which waits forever (resulting in a idle system). I've added a try/catch involving mapReduceCapable.executeMapReduceTask() and the TimeoutException still occurs but the master kill the slaves when the error ack is received.
My opinion: this is not a problem with radargun and not with Infinispan, but I'm going to wait for a second opinion.
In addition, I saw something strange: the cache is populated with Chinese chars instead of the "normal" chars. I believe it is due to the split of the file (or it is my dummy file). This creates a key's value with no spaces. Example:
PutKeyValueCommand{key=0-0, value=䉵琠䤠浵獴<U+20 ...
My dummy file (580 MB):
bigtext.txt: ASCII English text, with very long lines
> org.jgroups.TimeoutException after invoking MapCombineCommand in Map/Reduce task with 2 nodes
> ---------------------------------------------------------------------------------------------
>
> Key: ISPN-2836
> URL: https://issues.jboss.org/browse/ISPN-2836
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.2.1.Final
> Reporter: Alan Field
> Assignee: Pedro Ruivo
> Priority: Blocker
> Labels: onboard
> Fix For: 5.3.0.Final
>
> Attachments: afield-tcp-521-final.txt, benchmark-mapreduce-multifilesize.xml, dist-udp-no-tx.xml, jgroups-udp.xml, udp-edg-perf01.txt, udp-edg-perf02.txt
>
>
> Using RadarGun and two nodes to execute the example WordCount Map/Reduce job against a cache with ~550 keys with a value size of 1MB is producing a thread deadlock. The cache is distributed with transactions disabled.
> TCP transport deadlocks without throwing an exception. Disabling the send queue and setting UNICAST2.conn_expiry_timeout=0 prevents the deadlock, but the job does not complete. The nodes send "are-you-alive" messages back and forth, and I have seen the following exception:
> {noformat}
> 11:44:29,970 ERROR [org.jgroups.protocols.TCP] (OOB-98,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (76 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:352)
> at org.radargun.cachewrappers.InfinispanMapReduceWrapper.executeMapReduceTask(InfinispanMapReduceWrapper.java:98)
> at org.radargun.stages.MapReduceStage.executeOnSlave(MapReduceStage.java:74)
> at org.radargun.Slave$2.run(Slave.java:103)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.concurrent.ExecutionException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.infinispan.distexec.mapreduce.MapReduceTask$TaskPart.get(MapReduceTask.java:832)
> at org.infinispan.distexec.mapreduce.MapReduceTask.executeMapPhaseWithLocalReduction(MapReduceTask.java:477)
> at org.infinispan.distexec.mapreduce.MapReduceTask.execute(MapReduceTask.java:350)
> ... 9 more
> Caused by: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at org.infinispan.util.Util.rewrapAsCacheException(Util.java:541)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:186)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.invokeRemotely(JGroupsTransport.java:515)
> 11:44:29,978 ERROR [org.jgroups.protocols.TCP] (Timer-3,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (60 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:175)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:197)
> at org.infinispan.remoting.rpc.RpcManagerImpl.invokeRemotely(RpcManagerImpl.java:254)
> at org.infinispan.remoting.rpc.RpcManagerImpl.access$000(RpcManagerImpl.java:80)
> at org.infinispan.remoting.rpc.RpcManagerImpl$1.call(RpcManagerImpl.java:288)
> ... 5 more
> Caused by: org.jgroups.TimeoutException: timeout sending message to edg-perf02-32536
> at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:390)
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.processSingleCall(CommandAwareRpcDispatcher.java:301)
> 11:44:29,979 ERROR [org.jgroups.protocols.TCP] (Timer-4,default,edg-perf01-1907) failed sending message to edg-perf02-32536 (63 bytes): java.net.SocketException: Socket closed, cause: null
> at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.invokeRemoteCommand(CommandAwareRpcDispatcher.java:179)
> ... 11 more
> {noformat}
> With UDP transport, both threads are deadlocked. I will attach thread dumps from runs using TCP and UDP transport.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 10 months
[JBoss JIRA] (ISPN-3173) Add a query operation over hotrod
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3173?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3173:
--------------------------------
Description:
HR request: [op_id] [payload].
The request payload is query specific (not defined in the HR protocol) and at this stage has the following format: [Q_TYPE] [QUERY_ST] [FIRST_INDEX] [PAGE_SIZE]. This format accommodates the remote query requirements as defined in ISPN-3169.
- Q_TYPE (protobuf's byte) and query identifier, 1 for JPAQL (this is the query type we'll support). In future we'll add different query types as well.
- QUERY_STRING (protobuf's string)- JPAQL string generated by the fluent API (ISPN-3169). Parameters are encoded in this string (vs being sent separately)
- FIRST_INDEX + PAGE_SIZE (protobuf's int)- used for paginating/iterating over the result set.
HR response: [HR_SUCESS_FLAG] [payload]
PAYLOAD = [NUM_EL] [PROJ_SIZE] [ELEMENTS]
- even though at this stage we don't support projections (see ISPN-3169) PROJ_SIZE is added for future compatibility when projection will be supported.
Note that the payload for both request and response should be marshalled with protobuf as this information is read/written over
was:
HR request: [op_id] [payload].
The request payload is query specific (not defined in the HR protocol) and at this stage has the following format: [Q_TYPE] [QUERY_ST] [FIRST_INDEX] [PAGE_SIZE].
Q_TYPE and query identifier, might be JPAQL (this is the query type we'll support). In future we'll add different query types as well.
HR response: [success] [payload]
> Add a query operation over hotrod
> ----------------------------------
>
> Key: ISPN-3173
> URL: https://issues.jboss.org/browse/ISPN-3173
> Project: Infinispan
> Issue Type: Feature Request
> Reporter: Mircea Markus
> Assignee: Mircea Markus
>
> HR request: [op_id] [payload].
> The request payload is query specific (not defined in the HR protocol) and at this stage has the following format: [Q_TYPE] [QUERY_ST] [FIRST_INDEX] [PAGE_SIZE]. This format accommodates the remote query requirements as defined in ISPN-3169.
> - Q_TYPE (protobuf's byte) and query identifier, 1 for JPAQL (this is the query type we'll support). In future we'll add different query types as well.
> - QUERY_STRING (protobuf's string)- JPAQL string generated by the fluent API (ISPN-3169). Parameters are encoded in this string (vs being sent separately)
> - FIRST_INDEX + PAGE_SIZE (protobuf's int)- used for paginating/iterating over the result set.
> HR response: [HR_SUCESS_FLAG] [payload]
> PAYLOAD = [NUM_EL] [PROJ_SIZE] [ELEMENTS]
> - even though at this stage we don't support projections (see ISPN-3169) PROJ_SIZE is added for future compatibility when projection will be supported.
> Note that the payload for both request and response should be marshalled with protobuf as this information is read/written over
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 10 months
[JBoss JIRA] (ISPN-3173) Add a query operation over java hotrod client
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3173?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3173:
--------------------------------
Summary: Add a query operation over java hotrod client (was: Add a query operation over hotrod )
> Add a query operation over java hotrod client
> ----------------------------------------------
>
> Key: ISPN-3173
> URL: https://issues.jboss.org/browse/ISPN-3173
> Project: Infinispan
> Issue Type: Feature Request
> Reporter: Mircea Markus
> Assignee: Mircea Markus
>
> HR request: [op_id] [payload].
> The request payload is query specific (not defined in the HR protocol) and at this stage has the following format: [Q_TYPE] [QUERY_ST] [FIRST_INDEX] [PAGE_SIZE]. This format accommodates the remote query requirements as defined in ISPN-3169.
> - Q_TYPE (protobuf's byte) and query identifier, 1 for JPAQL (this is the query type we'll support). In future we'll add different query types as well.
> - QUERY_STRING (protobuf's string)- JPAQL string generated by the fluent API (ISPN-3169). Parameters are encoded in this string (vs being sent separately)
> - FIRST_INDEX + PAGE_SIZE (protobuf's int)- used for paginating/iterating over the result set.
> HR response: [HR_SUCESS_FLAG] [payload]
> PAYLOAD = [NUM_EL] [PROJ_SIZE] [ELEMENTS]
> - even though at this stage we don't support projections (see ISPN-3169) PROJ_SIZE is added for future compatibility when projection will be supported.
> Note that the payload for both request and response should be marshalled with protobuf as this information is read/written over
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 10 months
[JBoss JIRA] (ISPN-3173) Add a query operation over java hotrod client
by Mircea Markus (JIRA)
[ https://issues.jboss.org/browse/ISPN-3173?page=com.atlassian.jira.plugin.... ]
Mircea Markus updated ISPN-3173:
--------------------------------
Description:
HR request: [op_id] [payload].
The request payload is query specific (not defined in the HR protocol) and at this stage has the following format: [Q_TYPE] [QUERY_ST] [FIRST_INDEX] [PAGE_SIZE]. This format accommodates the remote query requirements as defined in ISPN-3169.
- Q_TYPE (protobuf's byte) and query identifier, 1 for JPAQL (this is the query type we'll support). In future we'll add different query types as well.
- QUERY_STRING (protobuf's string)- JPAQL string generated by the fluent API (ISPN-3169). Parameters are encoded in this string (vs being sent separately)
- FIRST_INDEX + PAGE_SIZE (protobuf's int)- used for paginating/iterating over the result set.
HR response: [HR_SUCESS_FLAG] [payload]
PAYLOAD = [NUM_EL] [PROJ_SIZE] [ELEMENTS]
- even though at this stage we don't support projections (see ISPN-3169) PROJ_SIZE is added for future compatibility when projection will be supported.
Note that the payload for both request and response should be marshalled with protobuf as this information is read/written over multiple clients.
was:
HR request: [op_id] [payload].
The request payload is query specific (not defined in the HR protocol) and at this stage has the following format: [Q_TYPE] [QUERY_ST] [FIRST_INDEX] [PAGE_SIZE]. This format accommodates the remote query requirements as defined in ISPN-3169.
- Q_TYPE (protobuf's byte) and query identifier, 1 for JPAQL (this is the query type we'll support). In future we'll add different query types as well.
- QUERY_STRING (protobuf's string)- JPAQL string generated by the fluent API (ISPN-3169). Parameters are encoded in this string (vs being sent separately)
- FIRST_INDEX + PAGE_SIZE (protobuf's int)- used for paginating/iterating over the result set.
HR response: [HR_SUCESS_FLAG] [payload]
PAYLOAD = [NUM_EL] [PROJ_SIZE] [ELEMENTS]
- even though at this stage we don't support projections (see ISPN-3169) PROJ_SIZE is added for future compatibility when projection will be supported.
Note that the payload for both request and response should be marshalled with protobuf as this information is read/written over
> Add a query operation over java hotrod client
> ----------------------------------------------
>
> Key: ISPN-3173
> URL: https://issues.jboss.org/browse/ISPN-3173
> Project: Infinispan
> Issue Type: Feature Request
> Reporter: Mircea Markus
> Assignee: Mircea Markus
>
> HR request: [op_id] [payload].
> The request payload is query specific (not defined in the HR protocol) and at this stage has the following format: [Q_TYPE] [QUERY_ST] [FIRST_INDEX] [PAGE_SIZE]. This format accommodates the remote query requirements as defined in ISPN-3169.
> - Q_TYPE (protobuf's byte) and query identifier, 1 for JPAQL (this is the query type we'll support). In future we'll add different query types as well.
> - QUERY_STRING (protobuf's string)- JPAQL string generated by the fluent API (ISPN-3169). Parameters are encoded in this string (vs being sent separately)
> - FIRST_INDEX + PAGE_SIZE (protobuf's int)- used for paginating/iterating over the result set.
> HR response: [HR_SUCESS_FLAG] [payload]
> PAYLOAD = [NUM_EL] [PROJ_SIZE] [ELEMENTS]
> - even though at this stage we don't support projections (see ISPN-3169) PROJ_SIZE is added for future compatibility when projection will be supported.
> Note that the payload for both request and response should be marshalled with protobuf as this information is read/written over multiple clients.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 10 months