[JBoss JIRA] (ISPN-2771) WriteSkewTest.testWriteSkewWithOnlyPut fails randomly
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2771?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2771:
-----------------------------------------------
Richard Janík <rjanik(a)redhat.com> made a comment on [bug 921491|https://bugzilla.redhat.com/show_bug.cgi?id=921491]
Test org.infinispan.api.mvcc.repeatable_read.WriteSkewTest.testWriteSkewWithOnlyPut fails intermittently with ER3 and Infinispan 5.2.5.Final as well.
> WriteSkewTest.testWriteSkewWithOnlyPut fails randomly
> -----------------------------------------------------
>
> Key: ISPN-2771
> URL: https://issues.jboss.org/browse/ISPN-2771
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite, Transactions
> Affects Versions: 5.2.0.CR3
> Reporter: Adrian Nistor
> Assignee: Adrian Nistor
> Fix For: 5.3.0.Alpha1
>
> Attachments: WriteSkewTest.zip
>
>
> {noformat}
> org.infinispan.api.mvcc.repeatable_read.WriteSkewTest:testWriteSkewWithOnlyPut
> org.infinispan.transaction.WriteSkewException: Detected write skew.
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> at org.infinispan.api.mvcc.repeatable_read.WriteSkewTest.testWriteSkewWithOnlyPut(WriteSkewTest.java:315)
> 22 lines not shown
> Caused by Detected write skew.
> org.infinispan.container.entries.RepeatableReadEntry.performLocalWriteSkewCheck(RepeatableReadEntry.java:68)
> at org.infinispan.container.entries.RepeatableReadEntry.copyForUpdate(RepeatableReadEntry.java:52)
> at org.infinispan.container.EntryFactoryImpl.wrapEntryForPut(EntryFactoryImpl.java:170)
> at org.infinispan.interceptors.EntryWrappingInterceptor.visitPutKeyValueCommand(EntryWrappingInterceptor.java:164)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.locking.OptimisticLockingInterceptor.visitPutKeyValueCommand(OptimisticLockingInterceptor.java:142)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:132)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.TxInterceptor.enlistWriteAndInvokeNext(TxInterceptor.java:251)
> at org.infinispan.interceptors.TxInterceptor.visitPutKeyValueCommand(TxInterceptor.java:191)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:118)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
> at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
> at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:62)
> at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:77)
> at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
> at org.infinispan.CacheImpl.executeCommandAndCommitIfNeeded(CacheImpl.java:1162)
> at org.infinispan.CacheImpl.putInternal(CacheImpl.java:760)
> at org.infinispan.CacheImpl.put(CacheImpl.java:754)
> at org.infinispan.CacheImpl.put(CacheImpl.java:748)
> at org.infinispan.CacheSupport.put(CacheSupport.java:53)
> at org.infinispan.api.mvcc.repeatable_read.WriteSkewTest$EntryWriter.call(WriteSkewTest.java:433)
> at org.infinispan.api.mvcc.repeatable_read.WriteSkewTest$EntryWriter.call(WriteSkewTest.java:418)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> 1 lines not shown
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2976) Log4J dependencies in codebase to be cleaned up
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-2976?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-2976:
------------------------------------
The Appender and Filter interfaces changed quite a bit, supporting both is going to be tricky. Log4j2 does include the Log4j 1.2 API in the bridge jar, but AFAICT it can't use Appenders defined with the old interface.
I made the minimal changes to make it work in this branch: https://github.com/danberindei/infinispan/tree/t_2976_m
> Log4J dependencies in codebase to be cleaned up
> -----------------------------------------------
>
> Key: ISPN-2976
> URL: https://issues.jboss.org/browse/ISPN-2976
> Project: Infinispan
> Issue Type: Task
> Affects Versions: 5.2.5.Final
> Reporter: Manik Surtani
> Assignee: Mircea Markus
> Fix For: 5.3.0.Alpha1, 5.3.0.Final
>
>
> When attempting to move to Log4J 2.0, I've noticed a number of hard deps on log4j classes.
> {{SampleConfigFilesCorrectnessTest}} - this class makes use of a custom appender to analyse what a user is being warned of when a config file is parsed. Why are we using Log4J for this? Our own logging interface should be mocked and messages captured directly.
> {{RehashStressTest}} and {{NucleotideCache}} - seems like a bug, I presume the author intended to use {{org.infinispan.logging.Log}}.
> {{CompressedFileAppender}} and {{ThreadNameFilter}}- can this be written in a way that works with Log4J 1.x as well as 2.x? Or have the SPIs changed that much?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2986) Intermittent failure to start new nodes during heavy write load
by Marc Bridner (JIRA)
[ https://issues.jboss.org/browse/ISPN-2986?page=com.atlassian.jira.plugin.... ]
Marc Bridner updated ISPN-2986:
-------------------------------
Description:
When under heavy write load from a hotrod client with 64+ threads and a new node is started, the new node will sometimes fail to start, eventually giving off state transfer timeouts and finally terminating. During the time it takes it to time out (~10 minutes) the hotrod client is totally blocked.
Setup is as follows:
3 servers, 1 client
* dl30x2385, 10.64.106.21, client
* dl30x2384, 10.64.106.20, first node
* dl30x2383, 10.64.106.19, second node
* dl30x2382, 10.64.106.18, third node
2 caches, initial state transfer off, transactions on, config is attached.
Small app that triggers the problem is also attached.
Steps to reproduce:
1. Start first node
2. Start client, wait for counter to reach 50000 (in client)
3. Start second node. 10% chance it'll fail.
4. Wait for counter to reach 100000 in client.
5. Start third node, 50% chance it'll fail.
If it doesn't fail, terminate everything and start over.
I realize this may be hard to reproduce, so if any more logs or tests are needed, let me know.
I've been unable to reproduce it on a single physical machine, and it only occurs when using more than 64 client threads. Changing the ratio of writes between the caches also seems to make it not occur. I was unable to reproduce it with TRACE log level on (too slow), but if you can specify some packages that you want traces of, that might work.
Turning transactions off makes it worse, 90% chance to fail on second node. Funny enough, disabling the concurrent GC lowers the failure rate to 10% on third node. Guessing race condition somewhere, may be similar to ISPN-2982.
was:
When under heavy write load from a hotrod client with 64+ threads and a new node is started, the new node will sometimes fail to start, eventually giving off state transfer timeouts and finally terminating. During the time it takes it to time out (~10 minutes) the hotrod client is totally blocked.
Setup is as follows:
3 servers, 1 client
* dl30x2385, 10.64.106.21, client
* dl30x2384, 10.64.106.20, first node
* dl30x2383, 10.64.106.19, second node
* dl30x2382, 10.64.106.18, third node
2 caches, initial state transfer off, transactions on, config is attached.
Small app that triggers the problem is also attached.
Steps to reproduce:
1. Start first node
2. Start client, wait for counter to reach 50000 (in client)
3. Start second node. 10% chance it'll fail.
4. Wait for counter to reach 100000 in client.
5. Start third node, 50% chance it'll fail.
If it doesn't fail, terminate everything and start over.
I realize this may be hard to reproduce, so if any more logs or tests are needed, let me know.
I've been unable to reproduce it on a single physical machine, and it only occurs when using more than 64 threads. Changing the ratio of writes between the caches also seems to make it not occur.
I was unable to reproduce it with TRACE log level on (too slow), but if you can specify some packages that you want traces of, that might work.
Turning transactions off makes it worse, 90% chance to fail on second node.
Funny enough, disabling the concurrent GC lowers the failure rate to 10% on third node. Guessing race condition somewhere, may be similar to ISPN-2982.
> Intermittent failure to start new nodes during heavy write load
> ---------------------------------------------------------------
>
> Key: ISPN-2986
> URL: https://issues.jboss.org/browse/ISPN-2986
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache, Server
> Affects Versions: 5.2.5.Final
> Environment: 4 servers running linux 2.6.32-220.13.1.el6.x86_64 with 2x QuadCore 2.4ghz CPU's
> Gigabit ethernet, same switch.
> java version "1.7.0"
> Java(TM) SE Runtime Environment (build 1.7.0-b147)
> Java HotSpot(TM) 64-Bit Server VM (build 21.0-b17, mixed mode)
> Reporter: Marc Bridner
> Assignee: Tristan Tarrant
> Attachments: logs.zip, test-infinispan.xml, test-jgroups.xml, test.infinispan.zip
>
>
> When under heavy write load from a hotrod client with 64+ threads and a new node is started, the new node will sometimes fail to start, eventually giving off state transfer timeouts and finally terminating. During the time it takes it to time out (~10 minutes) the hotrod client is totally blocked.
> Setup is as follows:
> 3 servers, 1 client
> * dl30x2385, 10.64.106.21, client
> * dl30x2384, 10.64.106.20, first node
> * dl30x2383, 10.64.106.19, second node
> * dl30x2382, 10.64.106.18, third node
> 2 caches, initial state transfer off, transactions on, config is attached.
> Small app that triggers the problem is also attached.
> Steps to reproduce:
> 1. Start first node
> 2. Start client, wait for counter to reach 50000 (in client)
> 3. Start second node. 10% chance it'll fail.
> 4. Wait for counter to reach 100000 in client.
> 5. Start third node, 50% chance it'll fail.
> If it doesn't fail, terminate everything and start over.
> I realize this may be hard to reproduce, so if any more logs or tests are needed, let me know.
> I've been unable to reproduce it on a single physical machine, and it only occurs when using more than 64 client threads. Changing the ratio of writes between the caches also seems to make it not occur. I was unable to reproduce it with TRACE log level on (too slow), but if you can specify some packages that you want traces of, that might work.
> Turning transactions off makes it worse, 90% chance to fail on second node. Funny enough, disabling the concurrent GC lowers the failure rate to 10% on third node. Guessing race condition somewhere, may be similar to ISPN-2982.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2986) Intermittent failure to start new nodes during heavy write load
by Marc Bridner (JIRA)
Marc Bridner created ISPN-2986:
----------------------------------
Summary: Intermittent failure to start new nodes during heavy write load
Key: ISPN-2986
URL: https://issues.jboss.org/browse/ISPN-2986
Project: Infinispan
Issue Type: Bug
Components: Distributed Cache, Server
Affects Versions: 5.2.5.Final
Environment: 4 servers running linux 2.6.32-220.13.1.el6.x86_64 with 2x QuadCore 2.4ghz CPU's
Gigabit ethernet, same switch.
java version "1.7.0"
Java(TM) SE Runtime Environment (build 1.7.0-b147)
Java HotSpot(TM) 64-Bit Server VM (build 21.0-b17, mixed mode)
Reporter: Marc Bridner
Assignee: Tristan Tarrant
Attachments: logs.zip, test-infinispan.xml, test-jgroups.xml, test.infinispan.zip
When under heavy write load from a hotrod client with 64+ threads and a new node is started, the new node will sometimes fail to start, eventually giving off state transfer timeouts and finally terminating. During the time it takes it to time out (~10 minutes) the hotrod client is totally blocked.
Setup is as follows:
3 servers, 1 client
* dl30x2385, 10.64.106.21, client
* dl30x2384, 10.64.106.20, first node
* dl30x2383, 10.64.106.19, second node
* dl30x2382, 10.64.106.18, third node
2 caches, initial state transfer off, transactions on, config is attached.
Small app that triggers the problem is also attached.
Steps to reproduce:
1. Start first node
2. Start client, wait for counter to reach 50000 (in client)
3. Start second node. 10% chance it'll fail.
4. Wait for counter to reach 100000 in client.
5. Start third node, 50% chance it'll fail.
If it doesn't fail, terminate everything and start over.
I realize this may be hard to reproduce, so if any more logs or tests are needed, let me know.
I've been unable to reproduce it on a single physical machine, and it only occurs when using more than 64 threads. Changing the ratio of writes between the caches also seems to make it not occur.
I was unable to reproduce it with TRACE log level on (too slow), but if you can specify some packages that you want traces of, that might work.
Turning transactions off makes it worse, 90% chance to fail on second node.
Funny enough, disabling the concurrent GC lowers the failure rate to 10% on third node. Guessing race condition somewhere, may be similar to ISPN-2982.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2986) Intermittent failure to start new nodes during heavy write load
by Marc Bridner (JIRA)
[ https://issues.jboss.org/browse/ISPN-2986?page=com.atlassian.jira.plugin.... ]
Marc Bridner updated ISPN-2986:
-------------------------------
Attachment: test.infinispan.zip
test-infinispan.xml
test-jgroups.xml
Config and code
> Intermittent failure to start new nodes during heavy write load
> ---------------------------------------------------------------
>
> Key: ISPN-2986
> URL: https://issues.jboss.org/browse/ISPN-2986
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache, Server
> Affects Versions: 5.2.5.Final
> Environment: 4 servers running linux 2.6.32-220.13.1.el6.x86_64 with 2x QuadCore 2.4ghz CPU's
> Gigabit ethernet, same switch.
> java version "1.7.0"
> Java(TM) SE Runtime Environment (build 1.7.0-b147)
> Java HotSpot(TM) 64-Bit Server VM (build 21.0-b17, mixed mode)
> Reporter: Marc Bridner
> Assignee: Tristan Tarrant
> Attachments: logs.zip, test-infinispan.xml, test-jgroups.xml, test.infinispan.zip
>
>
> When under heavy write load from a hotrod client with 64+ threads and a new node is started, the new node will sometimes fail to start, eventually giving off state transfer timeouts and finally terminating. During the time it takes it to time out (~10 minutes) the hotrod client is totally blocked.
> Setup is as follows:
> 3 servers, 1 client
> * dl30x2385, 10.64.106.21, client
> * dl30x2384, 10.64.106.20, first node
> * dl30x2383, 10.64.106.19, second node
> * dl30x2382, 10.64.106.18, third node
> 2 caches, initial state transfer off, transactions on, config is attached.
> Small app that triggers the problem is also attached.
> Steps to reproduce:
> 1. Start first node
> 2. Start client, wait for counter to reach 50000 (in client)
> 3. Start second node. 10% chance it'll fail.
> 4. Wait for counter to reach 100000 in client.
> 5. Start third node, 50% chance it'll fail.
> If it doesn't fail, terminate everything and start over.
> I realize this may be hard to reproduce, so if any more logs or tests are needed, let me know.
> I've been unable to reproduce it on a single physical machine, and it only occurs when using more than 64 threads. Changing the ratio of writes between the caches also seems to make it not occur.
> I was unable to reproduce it with TRACE log level on (too slow), but if you can specify some packages that you want traces of, that might work.
> Turning transactions off makes it worse, 90% chance to fail on second node.
> Funny enough, disabling the concurrent GC lowers the failure rate to 10% on third node. Guessing race condition somewhere, may be similar to ISPN-2982.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2986) Intermittent failure to start new nodes during heavy write load
by Marc Bridner (JIRA)
[ https://issues.jboss.org/browse/ISPN-2986?page=com.atlassian.jira.plugin.... ]
Marc Bridner updated ISPN-2986:
-------------------------------
Attachment: logs.zip
Logs and stack traces
> Intermittent failure to start new nodes during heavy write load
> ---------------------------------------------------------------
>
> Key: ISPN-2986
> URL: https://issues.jboss.org/browse/ISPN-2986
> Project: Infinispan
> Issue Type: Bug
> Components: Distributed Cache, Server
> Affects Versions: 5.2.5.Final
> Environment: 4 servers running linux 2.6.32-220.13.1.el6.x86_64 with 2x QuadCore 2.4ghz CPU's
> Gigabit ethernet, same switch.
> java version "1.7.0"
> Java(TM) SE Runtime Environment (build 1.7.0-b147)
> Java HotSpot(TM) 64-Bit Server VM (build 21.0-b17, mixed mode)
> Reporter: Marc Bridner
> Assignee: Tristan Tarrant
> Attachments: logs.zip, test-infinispan.xml, test-jgroups.xml, test.infinispan.zip
>
>
> When under heavy write load from a hotrod client with 64+ threads and a new node is started, the new node will sometimes fail to start, eventually giving off state transfer timeouts and finally terminating. During the time it takes it to time out (~10 minutes) the hotrod client is totally blocked.
> Setup is as follows:
> 3 servers, 1 client
> * dl30x2385, 10.64.106.21, client
> * dl30x2384, 10.64.106.20, first node
> * dl30x2383, 10.64.106.19, second node
> * dl30x2382, 10.64.106.18, third node
> 2 caches, initial state transfer off, transactions on, config is attached.
> Small app that triggers the problem is also attached.
> Steps to reproduce:
> 1. Start first node
> 2. Start client, wait for counter to reach 50000 (in client)
> 3. Start second node. 10% chance it'll fail.
> 4. Wait for counter to reach 100000 in client.
> 5. Start third node, 50% chance it'll fail.
> If it doesn't fail, terminate everything and start over.
> I realize this may be hard to reproduce, so if any more logs or tests are needed, let me know.
> I've been unable to reproduce it on a single physical machine, and it only occurs when using more than 64 threads. Changing the ratio of writes between the caches also seems to make it not occur.
> I was unable to reproduce it with TRACE log level on (too slow), but if you can specify some packages that you want traces of, that might work.
> Turning transactions off makes it worse, 90% chance to fail on second node.
> Funny enough, disabling the concurrent GC lowers the failure rate to 10% on third node. Guessing race condition somewhere, may be similar to ISPN-2982.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years
[JBoss JIRA] (ISPN-2985) HotRod endpoint SSL support
by Tristan Tarrant (JIRA)
Tristan Tarrant created ISPN-2985:
-------------------------------------
Summary: HotRod endpoint SSL support
Key: ISPN-2985
URL: https://issues.jboss.org/browse/ISPN-2985
Project: Infinispan
Issue Type: Feature Request
Components: Server
Reporter: Tristan Tarrant
Assignee: Tristan Tarrant
Fix For: 5.3.0.Final
The server endpoints should be able to use a security-domain for SSL configuration
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
11 years