[JBoss JIRA] (ISPN-9530) Error marshalling events with NearCache
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-9530?page=com.atlassian.jira.plugin.... ]
William Burns updated ISPN-9530:
--------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 9.4.0.Final
Resolution: Done
> Error marshalling events with NearCache
> ---------------------------------------
>
> Key: ISPN-9530
> URL: https://issues.jboss.org/browse/ISPN-9530
> Project: Infinispan
> Issue Type: Bug
> Components: Listeners
> Reporter: William Burns
> Assignee: William Burns
> Fix For: 9.4.0.Final
>
>
> The test https://ci.infinispan.org/job/Infinispan/job/PR-6232/17/testReport/junit/... seems to fail randomly. It is caused due to the fact that the event cannot be marshalled properly.
> The following is the stack trace when the exception is met
> {code}
> 09:11:00,159 TRACE (HotRod-client-async-pool-3:[]) [HeaderDecoder] Response 23 belongs to PutOperation{(default), key=[B0x034BEBCEF4CA, value=[B0x033E027632, flags=6} on [id: 0x876415fe, L:/127.0.0.1:41108 - R:127.0.0.1/127.0.0.1:34808]
> 09:11:00,159 TRACE (HotRod-client-async-pool-3:[]) [HeaderDecoder] Decoding header for PutOperation{(default), key=[B0x034BEBCEF4CA, value=[B0x033E027632, flags=6} on [id: 0x876415fe, L:/127.0.0.1:41108 - R:127.0.0.1/127.0.0.1:34808]
> 09:11:00,159 WARN (HotRod-client-async-pool-2:[]) [HeaderDecoder] ISPN004039: Unable to complete reading event from server 127.0.0.1/127.0.0.1:39088
> org.infinispan.client.hotrod.exceptions.HotRodClientException: ISPN004034: Unable to unmarshall bytes 3C737472696E673E262378333B4BEFBFBDEFBFBDEFBFBDEFBFBD3C2F737472696E673E
> at org.infinispan.client.hotrod.marshall.MarshallerUtil.bytes2obj(MarshallerUtil.java:47) ~[classes/:?]
> at org.infinispan.client.hotrod.DataFormat.keyToObj(DataFormat.java:103) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.protocol.Codec21.readCacheEvent(Codec21.java:79) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.HeaderDecoder.decode(HeaderDecoder.java:153) [classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.HintedReplayingDecoder.callDecode(HintedReplayingDecoder.java:98) [classes/:?]
> at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.22.Final.jar:4.1.22.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.22.Final.jar:4.1.22.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.22.Final.jar:4.1.22.Final]
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.22.Final.jar:4.1.22.Final]
> at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.22.Final.jar:4.1.22.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.22.Final.jar:4.1.22.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.22.Final.jar:4.1.22.Final]
> at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.22.Final.jar:4.1.22.Final]
> at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1414) [netty-transport-4.1.22.Final.jar:4.1.22.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.22.Final.jar:4.1.22.Final]
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.22.Final.jar:4.1.22.Final]
> at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:945) [netty-transport-4.1.22.Final.jar:4.1.22.Final]
> at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:806) [netty-transport-native-epoll-4.1.22.Final-linux-x86_64.jar:4.1.22.Final]
> at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:404) [netty-transport-native-epoll-4.1.22.Final-linux-x86_64.jar:4.1.22.Final]
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:304) [netty-transport-native-epoll-4.1.22.Final-linux-x86_64.jar:4.1.22.Final]
> at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) [netty-common-4.1.22.Final.jar:4.1.22.Final]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
> Caused by: java.io.IOException: Unsupported protocol version 60
> at org.jboss.marshalling.river.RiverUnmarshaller.start(RiverUnmarshaller.java:1345) ~[jboss-marshalling-osgi-2.0.5.Final.jar:2.0.5.Final]
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.startObjectInput(AbstractJBossMarshaller.java:129) ~[classes/:?]
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromByteBuffer(AbstractJBossMarshaller.java:110) ~[classes/:?]
> at org.infinispan.commons.marshall.AbstractMarshaller.objectFromByteBuffer(AbstractMarshaller.java:82) ~[classes/:?]
> at org.infinispan.client.hotrod.marshall.MarshallerUtil.bytes2obj(MarshallerUtil.java:31) ~[classes/:?]
> ... 23 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years
[JBoss JIRA] (ISPN-9195) Iterator leak when Hot Rod client crashes
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-9195?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes commented on ISPN-9195:
-----------------------------------------
[~william.burns] I agree it makes sense implementing it in core first. Do you want to take this JIRA?
> Iterator leak when Hot Rod client crashes
> -----------------------------------------
>
> Key: ISPN-9195
> URL: https://issues.jboss.org/browse/ISPN-9195
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 9.3.0.Beta1
> Reporter: Radim Vansa
> Assignee: Gustavo Fernandes
> Labels: memory_leak
>
> When the client abruptly terminates, or simply does not close the iterator, iterator resources on server are not released.
> Any such resources should be handles with lease-based expiration. Additionally we could remove the resources when the connection that established them is closed.
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years
[JBoss JIRA] (ISPN-9127) Remote commands can access components before they are started
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-9127?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes commented on ISPN-9127:
-----------------------------------------
I attached server0, server1 and server2, they're about a deadlock in the state transfer when starting 3 server sequentially
Also attached trace.tar.gz, which is also about the state transfer deadlock but not the same execution as above.
> Remote commands can access components before they are started
> -------------------------------------------------------------
>
> Key: ISPN-9127
> URL: https://issues.jboss.org/browse/ISPN-9127
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.2.2.Final, 9.3.0.Alpha1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Attachments: server0, server1, server2, trace.tar.gz
>
>
> {{PerCacheInboundInvocationHandler.handle()}} may be called before the component was started, because {{GlobalInboundInvocationHandler}} fetches it from the component registry without any checks. {{CommandsFactoryImpl.initializeReplicableCommand()}} doesn't wait for the components that it injects into remote commands to be started, either.
> This started causing random test failures in {{ConcurrentStartForkChannelTest}} after ISPN-8515, which moved most initialization work from {{init()}} methods to {{start()}} methods. Because {{StateProviderImpl}} starts after {{StateTransferManagerImpl}}, it's possible for a node to receive a {{StateRequestCommand}} before {{StateProviderImpl}} has initialized:
> {noformat}
> 16:15:09,549 TRACE (remote-thread-Test-NodeB-p51957-t2:[org.infinispan.CONFIG]) [StateProviderImpl] Starting outbound transfer to node Test-NodeA for cache null, topology id 2, segments {0-255}
> 16:15:09,551 WARN (remote-thread-Test-NodeB-p51957-t2:[]) [NonTotalOrderPerCacheInboundInvocationHandler] ISPN000071: Caught exception when handling command StateRequestCommand{cache=org.infinispan.CONFIG, origin=Test-NodeA, type=START_STATE_TRANSFER, topologyId=2, segments={0-255}}
> java.lang.IllegalArgumentException: chunkSize must be greater than 0
> at org.infinispan.statetransfer.OutboundTransferTask.<init>(OutboundTransferTask.java:114) ~[classes/:?]
> at org.infinispan.statetransfer.StateProviderImpl.startOutboundTransfer(StateProviderImpl.java:273) ~[classes/:?]
> at org.infinispan.statetransfer.StateRequestCommand.invokeAsync(StateRequestCommand.java:101) ~[classes/:?]
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94) ~[classes/:?]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years
[JBoss JIRA] (ISPN-9127) Remote commands can access components before they are started
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-9127?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-9127:
------------------------------------
Attachment: trace.tar.gz
> Remote commands can access components before they are started
> -------------------------------------------------------------
>
> Key: ISPN-9127
> URL: https://issues.jboss.org/browse/ISPN-9127
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.2.2.Final, 9.3.0.Alpha1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Attachments: server0, server1, server2, trace.tar.gz
>
>
> {{PerCacheInboundInvocationHandler.handle()}} may be called before the component was started, because {{GlobalInboundInvocationHandler}} fetches it from the component registry without any checks. {{CommandsFactoryImpl.initializeReplicableCommand()}} doesn't wait for the components that it injects into remote commands to be started, either.
> This started causing random test failures in {{ConcurrentStartForkChannelTest}} after ISPN-8515, which moved most initialization work from {{init()}} methods to {{start()}} methods. Because {{StateProviderImpl}} starts after {{StateTransferManagerImpl}}, it's possible for a node to receive a {{StateRequestCommand}} before {{StateProviderImpl}} has initialized:
> {noformat}
> 16:15:09,549 TRACE (remote-thread-Test-NodeB-p51957-t2:[org.infinispan.CONFIG]) [StateProviderImpl] Starting outbound transfer to node Test-NodeA for cache null, topology id 2, segments {0-255}
> 16:15:09,551 WARN (remote-thread-Test-NodeB-p51957-t2:[]) [NonTotalOrderPerCacheInboundInvocationHandler] ISPN000071: Caught exception when handling command StateRequestCommand{cache=org.infinispan.CONFIG, origin=Test-NodeA, type=START_STATE_TRANSFER, topologyId=2, segments={0-255}}
> java.lang.IllegalArgumentException: chunkSize must be greater than 0
> at org.infinispan.statetransfer.OutboundTransferTask.<init>(OutboundTransferTask.java:114) ~[classes/:?]
> at org.infinispan.statetransfer.StateProviderImpl.startOutboundTransfer(StateProviderImpl.java:273) ~[classes/:?]
> at org.infinispan.statetransfer.StateRequestCommand.invokeAsync(StateRequestCommand.java:101) ~[classes/:?]
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94) ~[classes/:?]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years
[JBoss JIRA] (ISPN-9127) Remote commands can access components before they are started
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-9127?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-9127:
------------------------------------
Attachment: server2
server1
server0
> Remote commands can access components before they are started
> -------------------------------------------------------------
>
> Key: ISPN-9127
> URL: https://issues.jboss.org/browse/ISPN-9127
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.2.2.Final, 9.3.0.Alpha1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Labels: testsuite_stability
> Attachments: server0, server1, server2
>
>
> {{PerCacheInboundInvocationHandler.handle()}} may be called before the component was started, because {{GlobalInboundInvocationHandler}} fetches it from the component registry without any checks. {{CommandsFactoryImpl.initializeReplicableCommand()}} doesn't wait for the components that it injects into remote commands to be started, either.
> This started causing random test failures in {{ConcurrentStartForkChannelTest}} after ISPN-8515, which moved most initialization work from {{init()}} methods to {{start()}} methods. Because {{StateProviderImpl}} starts after {{StateTransferManagerImpl}}, it's possible for a node to receive a {{StateRequestCommand}} before {{StateProviderImpl}} has initialized:
> {noformat}
> 16:15:09,549 TRACE (remote-thread-Test-NodeB-p51957-t2:[org.infinispan.CONFIG]) [StateProviderImpl] Starting outbound transfer to node Test-NodeA for cache null, topology id 2, segments {0-255}
> 16:15:09,551 WARN (remote-thread-Test-NodeB-p51957-t2:[]) [NonTotalOrderPerCacheInboundInvocationHandler] ISPN000071: Caught exception when handling command StateRequestCommand{cache=org.infinispan.CONFIG, origin=Test-NodeA, type=START_STATE_TRANSFER, topologyId=2, segments={0-255}}
> java.lang.IllegalArgumentException: chunkSize must be greater than 0
> at org.infinispan.statetransfer.OutboundTransferTask.<init>(OutboundTransferTask.java:114) ~[classes/:?]
> at org.infinispan.statetransfer.StateProviderImpl.startOutboundTransfer(StateProviderImpl.java:273) ~[classes/:?]
> at org.infinispan.statetransfer.StateRequestCommand.invokeAsync(StateRequestCommand.java:101) ~[classes/:?]
> at org.infinispan.remoting.inboundhandler.BasePerCacheInboundInvocationHandler.invokeCommand(BasePerCacheInboundInvocationHandler.java:94) ~[classes/:?]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
7 years