[JBoss JIRA] (ISPN-2697) HotRodServer startup fails when its record cannot be inserted into topology cache
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2697?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-2697:
------------------------------------
All the flags we have so far are related to cache operations, and are not relevant for state transfer commands. That's the reason state transfer commands to not carry any flags.
I'm not saying that we absolutely shouldn't use a flag for the HotRod server startup issue, I'm just saying it won't be in any way related to the way the RSVP flag is added for the state transfer commands.
OTOH I *am* saying that I don't see the reason for adding a new flag, when this can be handled much easier by adjusting the replTimeout and desired_avg_gossip settings.
> HotRodServer startup fails when its record cannot be inserted into topology cache
> ---------------------------------------------------------------------------------
>
> Key: ISPN-2697
> URL: https://issues.jboss.org/browse/ISPN-2697
> Project: Infinispan
> Issue Type: Bug
> Components: Remote protocols
> Affects Versions: 5.2.0.Beta6
> Reporter: Radim Vansa
> Assignee: Galder Zamarreño
> Priority: Critical
> Fix For: 5.2.0.Final
>
>
> When the HotRodServer starts it inserts its record to __hotRodTopologyCache ({{HotRodServer.addSelfToTopologyView(...)}}).
> However, this put may very easily fail - as the command is broadcasted using NAKACK2 protocol, if the message gets lost and there's no following broadcasted message, the message will be not retransmitted and the put operation times out (Replication timeout), which fails the whole HotRodServer startup, all because of one lost UDP message.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2717) execute(Runnable) of DistributedExecutorService doesn't show exception message in case of failure of Runnable
by Anna Manukyan (JIRA)
[ https://issues.jboss.org/browse/ISPN-2717?page=com.atlassian.jira.plugin.... ]
Anna Manukyan commented on ISPN-2717:
-------------------------------------
Ok, got it! I was running tests with other profile rather than the traceTests, that's why I couldn't see the logs. Thanks a lot and sorry for inconvenience.
> execute(Runnable) of DistributedExecutorService doesn't show exception message in case of failure of Runnable
> -------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2717
> URL: https://issues.jboss.org/browse/ISPN-2717
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.2.0.CR2
> Reporter: Anna Manukyan
> Assignee: Galder Zamarreño
>
> Assume, that we have a runnable which run() method throws RuntimeException.
> In case if the runnable is executed using void execute(Runnable command) for DistributedExecutorService, the execution of runnable fails - but there is no any stacktrace printed in the logs.
> So it is really hard to find, what was wrong in your Runnable.
> The sample test which does this is:
> {code}
> public void testRunnableExecution() throws InterruptedException {
> //assuming cache initialization is done
> DistributedExecutorService des = new DefaultExecutorService(cache);
> SampleRunnable runnable = new SampleRunnable();
> des.execute(runnable);
> }
> static class SampleRunnable implements Runnable, Serializable {
> public SampleRunnable() {
> }
> @Override
> public void run() {
> System.out.println("Running runnable");
> int i = 5 / 0;
> }
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2771) WriteSkewTest.testWriteSkewWithOnlyPut fails randomly
by Adrian Nistor (JIRA)
[ https://issues.jboss.org/browse/ISPN-2771?page=com.atlassian.jira.plugin.... ]
Adrian Nistor updated ISPN-2771:
--------------------------------
Attachment: WriteSkewTest.zip
Trace log for the whole test.
> WriteSkewTest.testWriteSkewWithOnlyPut fails randomly
> -----------------------------------------------------
>
> Key: ISPN-2771
> URL: https://issues.jboss.org/browse/ISPN-2771
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite, Transactions
> Affects Versions: 5.2.0.CR3
> Reporter: Adrian Nistor
> Assignee: Mircea Markus
> Fix For: 5.3.0.Beta1
>
> Attachments: WriteSkewTest.zip
>
>
> {noformat}
> org.infinispan.api.mvcc.repeatable_read.WriteSkewTest:testWriteSkewWithOnlyPut
> org.infinispan.transaction.WriteSkewException: Detected write skew.
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.infinispan.api.mvcc.repeatable_read.WriteSkewTest.testWriteSkewWithOnlyPut(WriteSkewTest.java:315)
> 22 lines not shown
> Caused by Detected write skew.
> org.infinispan.container.entries.RepeatableReadEntry.performLocalWriteSkewCheck(RepeatableReadEntry.java:68)
> at org.infinispan.container.entries.RepeatableReadEntry.copyForUpdate(RepeatableReadEntry.java:52)
> at org.infinispan.container.EntryFactoryImpl.wrapEntryForPut(EntryFactoryImpl.java:170)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:164)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:142)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:251)
> at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:191)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1162)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:760)
> at org.infinispan.CacheImpl.put(CacheImpl.java:754)
> at org.infinispan.CacheImpl.put(CacheImpl.java:748)
> at org.infinispan.CacheSupport.put(CacheSupport.java:53)
> at org.infinispan.api.mvcc.repeatable_read.WriteSkewTest$EntryWriter.call(WriteSkewTest.java:433)
> at org.infinispan.api.mvcc.repeatable_read.WriteSkewTest$EntryWriter.call(WriteSkewTest.java:418)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 1 lines not shown
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2771) WriteSkewTest.testWriteSkewWithOnlyPut fails randomly
by Adrian Nistor (JIRA)
Adrian Nistor created ISPN-2771:
-----------------------------------
Summary: WriteSkewTest.testWriteSkewWithOnlyPut fails randomly
Key: ISPN-2771
URL: https://issues.jboss.org/browse/ISPN-2771
Project: Infinispan
Issue Type: Bug
Components: Test Suite, Transactions
Affects Versions: 5.2.0.CR3
Reporter: Adrian Nistor
Assignee: Mircea Markus
Fix For: 5.3.0.Beta1
{noformat}
org.infinispan.api.mvcc.repeatable_read.WriteSkewTest:testWriteSkewWithOnlyPut
org.infinispan.transaction.WriteSkewException: Detected write skew.
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at org.infinispan.api.mvcc.repeatable_read.WriteSkewTest.testWriteSkewWithOnlyPut(WriteSkewTest.java:315)
22 lines not shown
Caused by Detected write skew.
org.infinispan.container.entries.RepeatableReadEntry.performLocalWriteSkewCheck(RepeatableReadEntry.java:68)
at org.infinispan.container.entries.RepeatableReadEntry.copyForUpdate(RepeatableReadEntry.java:52)
at org.infinispan.container.EntryFactoryImpl.wrapEntryForPut(EntryFactoryImpl.java:170)
at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:164)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:142)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:251)
at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:191)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1162)
at org.infinispan.CacheImpl.putInternal(CacheImpl.java:760)
at org.infinispan.CacheImpl.put(CacheImpl.java:754)
at org.infinispan.CacheImpl.put(CacheImpl.java:748)
at org.infinispan.CacheSupport.put(CacheSupport.java:53)
at org.infinispan.api.mvcc.repeatable_read.WriteSkewTest$EntryWriter.call(WriteSkewTest.java:433)
at org.infinispan.api.mvcc.repeatable_read.WriteSkewTest$EntryWriter.call(WriteSkewTest.java:418)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
1 lines not shown
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2699) Failed ping on startup can leave socket dirty
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-2699?page=com.atlassian.jira.plugin.... ]
Work on ISPN-2699 started by Galder Zamarreño.
> Failed ping on startup can leave socket dirty
> ---------------------------------------------
>
> Key: ISPN-2699
> URL: https://issues.jboss.org/browse/ISPN-2699
> Project: Infinispan
> Issue Type: Bug
> Components: Remote protocols
> Affects Versions: 5.2.0.Beta6
> Reporter: Radim Vansa
> Assignee: Galder Zamarreño
> Priority: Critical
> Fix For: 5.2.0.Final
>
>
> Ping on startup in TransportObjectFactory.makeObject does not check the return value of ping operation, neither the tcpTransport.isValid().
> Therefore, if a HotRodClientException is thrown when the response header is read from socket, the socket is left in a dirty state.
> In our case this resulted in reading response to previous request (the request was written but the response was not read due to some timeout and then, when another request was executed it read the response to the first request). This caused {{ISPN004004: Invalid message id. Expected 2930 and received 212}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2699) Failed ping on startup can leave socket dirty
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-2699?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-2699:
----------------------------------------
I see a couple of problems here:
1. We don't throw any exceptions out of makeObject() is there's any issues with ping() operation. That means both propagating exceptions and other failure conditions. There's no need to check tcpTransport.isValid() because no other thread has used that transport instance, so the socket can't be closed and can't be invalid before it's returned out of makeObject().
2. The server could potentially return a topology if there's any server side error, but the client won't read it unless there was success. This needs some thinking.
What could have happened here is that an exceptional status has been returned and the topology has not been read. To the pool, it would seem that the socket is valid and would therefore return it to another client. The client then sends the request and starts reading the response from the socket and the first thing it'd find is the topology header of the old socket, which would explain the message id mismatch.
> Failed ping on startup can leave socket dirty
> ---------------------------------------------
>
> Key: ISPN-2699
> URL: https://issues.jboss.org/browse/ISPN-2699
> Project: Infinispan
> Issue Type: Bug
> Components: Remote protocols
> Affects Versions: 5.2.0.Beta6
> Reporter: Radim Vansa
> Assignee: Galder Zamarreño
> Fix For: 5.2.0.Final
>
>
> Ping on startup in TransportObjectFactory.makeObject does not check the return value of ping operation, neither the tcpTransport.isValid().
> Therefore, if a HotRodClientException is thrown when the response header is read from socket, the socket is left in a dirty state.
> In our case this resulted in reading response to previous request (the request was written but the response was not read due to some timeout and then, when another request was executed it read the response to the first request). This caused {{ISPN004004: Invalid message id. Expected 2930 and received 212}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2699) Failed ping on startup can leave socket dirty
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-2699?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño updated ISPN-2699:
-----------------------------------
Priority: Critical (was: Major)
> Failed ping on startup can leave socket dirty
> ---------------------------------------------
>
> Key: ISPN-2699
> URL: https://issues.jboss.org/browse/ISPN-2699
> Project: Infinispan
> Issue Type: Bug
> Components: Remote protocols
> Affects Versions: 5.2.0.Beta6
> Reporter: Radim Vansa
> Assignee: Galder Zamarreño
> Priority: Critical
> Fix For: 5.2.0.Final
>
>
> Ping on startup in TransportObjectFactory.makeObject does not check the return value of ping operation, neither the tcpTransport.isValid().
> Therefore, if a HotRodClientException is thrown when the response header is read from socket, the socket is left in a dirty state.
> In our case this resulted in reading response to previous request (the request was written but the response was not read due to some timeout and then, when another request was executed it read the response to the first request). This caused {{ISPN004004: Invalid message id. Expected 2930 and received 212}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2717) execute(Runnable) of DistributedExecutorService doesn't show exception message in case of failure of Runnable
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-2717?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-2717:
----------------------------------------
Anna, are you sure about this? If the executable is invoked locally, any throwable should be logged:
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
Even if the execution is remote, logging happens:
https://github.com/infinispan/infinispan/blob/master/core/src/main/java/o...
That test does not wait for the execution to finish, so it could happen that the JVM stops before it's executed the code and it's printed the exception.
Rejecting this for the moment until more proof is provided on the existence of a bug.
> execute(Runnable) of DistributedExecutorService doesn't show exception message in case of failure of Runnable
> -------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2717
> URL: https://issues.jboss.org/browse/ISPN-2717
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.2.0.CR2
> Reporter: Anna Manukyan
> Assignee: Galder Zamarreño
> Fix For: 5.2.0.Final
>
>
> Assume, that we have a runnable which run() method throws RuntimeException.
> In case if the runnable is executed using void execute(Runnable command) for DistributedExecutorService, the execution of runnable fails - but there is no any stacktrace printed in the logs.
> So it is really hard to find, what was wrong in your Runnable.
> The sample test which does this is:
> {code}
> public void testRunnableExecution() throws InterruptedException {
> //assuming cache initialization is done
> DistributedExecutorService des = new DefaultExecutorService(cache);
> SampleRunnable runnable = new SampleRunnable();
> des.execute(runnable);
> }
> static class SampleRunnable implements Runnable, Serializable {
> public SampleRunnable() {
> }
> @Override
> public void run() {
> System.out.println("Running runnable");
> int i = 5 / 0;
> }
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months
[JBoss JIRA] (ISPN-2717) execute(Runnable) of DistributedExecutorService doesn't show exception message in case of failure of Runnable
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-2717?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño resolved ISPN-2717.
------------------------------------
Fix Version/s: (was: 5.2.0.Final)
Resolution: Rejected
> execute(Runnable) of DistributedExecutorService doesn't show exception message in case of failure of Runnable
> -------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2717
> URL: https://issues.jboss.org/browse/ISPN-2717
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Execution and Map/Reduce
> Affects Versions: 5.2.0.CR2
> Reporter: Anna Manukyan
> Assignee: Galder Zamarreño
>
> Assume, that we have a runnable which run() method throws RuntimeException.
> In case if the runnable is executed using void execute(Runnable command) for DistributedExecutorService, the execution of runnable fails - but there is no any stacktrace printed in the logs.
> So it is really hard to find, what was wrong in your Runnable.
> The sample test which does this is:
> {code}
> public void testRunnableExecution() throws InterruptedException {
> //assuming cache initialization is done
> DistributedExecutorService des = new DefaultExecutorService(cache);
> SampleRunnable runnable = new SampleRunnable();
> des.execute(runnable);
> }
> static class SampleRunnable implements Runnable, Serializable {
> public SampleRunnable() {
> }
> @Override
> public void run() {
> System.out.println("Running runnable");
> int i = 5 / 0;
> }
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years, 11 months