[JBoss JIRA] (ISPN-5123) MultiNodeDistributedTest deadlock
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-5123?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-5123:
------------------------------------
Attachment: deadlock_sep_10.txt
> MultiNodeDistributedTest deadlock
> ---------------------------------
>
> Key: ISPN-5123
> URL: https://issues.jboss.org/browse/ISPN-5123
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Query
> Affects Versions: 7.1.0.Alpha1
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Attachments: deadlock_sep_10.txt, infinispan-infinispan-query.log, trace.tar.gz
>
>
> I've been seeing this intermittent problem in my environment. Sometimes the query suite hangs for 30min (and then proceeds). See attached stack trace.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 6 months
[JBoss JIRA] (ISPN-5123) MultiNodeDistributedTest deadlock
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-5123?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes commented on ISPN-5123:
-----------------------------------------
Last stack trace with the deadlock shows several threads waiting for the view and another trying to stop the component registry:
{noformat}
"testng-MultiNodeReplicatedTest" #11 prio=5 os_prio=0 tid=0x00007fcf0c87e000 nid=0x3276 waiting on condition [0x00007fcee18e7000]
- locked <0x0000000087b59758> (a org.infinispan.factories.ComponentRegistry)
"OOB-3,MultiNodeReplicatedTest-NodeC-40758" #45486 prio=5 os_prio=0 tid=0x00007fcecc5ac800 nid=0x6636 waiting on condition [0x00007fce98205000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000087b41f18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.waitForView(JGroupsTransport.java:260)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.waitForView(JGroupsTransport.java:248)
at org.infinispan.topology.LocalTopologyManagerImpl.waitForView(LocalTopologyManagerImpl.java:545)
"OOB-6,MultiNodeReplicatedTest-NodeC-40758" #89735 prio=5 os_prio=0 tid=0x00007fcecc45c800 nid=0x16c7 waiting on condition [0x00007fcead85a000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000087b41f18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.waitForView(JGroupsTransport.java:260)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.waitForView(JGroupsTransport.java:248)
at org.infinispan.topology.LocalTopologyManagerImpl.waitForView(LocalTopologyManagerImpl.java:545)
{noformat}
> MultiNodeDistributedTest deadlock
> ---------------------------------
>
> Key: ISPN-5123
> URL: https://issues.jboss.org/browse/ISPN-5123
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Query
> Affects Versions: 7.1.0.Alpha1
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
> Attachments: infinispan-infinispan-query.log, trace.tar.gz
>
>
> I've been seeing this intermittent problem in my environment. Sometimes the query suite hangs for 30min (and then proceeds). See attached stack trace.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 6 months
[JBoss JIRA] (ISPN-2103) Concurrent access after removal of an AtomicMap should NOT result in an IllegalStateException when accessed by other threads
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-2103?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant reassigned ISPN-2103:
-------------------------------------
Assignee: Pedro Ruivo (was: Tristan Tarrant)
> Concurrent access after removal of an AtomicMap should NOT result in an IllegalStateException when accessed by other threads
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2103
> URL: https://issues.jboss.org/browse/ISPN-2103
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.1.5.FINAL
> Reporter: Adrian Nistor
> Assignee: Pedro Ruivo
>
> ISPN-1121 introduces an IllegalStateException that is being thrown from AtomicMap methods once the map handle has become stale (ie. removed from cache). In case of concurrent access, the exception is thrown to all threads not just to the thread that performed the removal. This was fine-ish in older versions of Infinispan before introduction of optimistic and pessimistic locking but it should be reconsidered now because:
> 1. It interferes/overlaps with transaction isolation. We should rely on the selected locking scheme (OL/PL) to detect conflicts between transactions and report the problem accordingly. Especially if the stale map is used just for reading - this should be allowed to work rather than stop it.
> 2. This exception is pretty disruptive and awkward to handle. All methods of an AtomicMap can result in this exception and the current thread has to be prepared for handling it if other threads remove the map. Not much transaction isolation.
> 3. Since the TreeCache is backed by AtomicMap nearly all Tree API can throw this.
> The proposed fix consists of:
> 1. removing AtomicHashMap.removed flag and AtomicHashMap.markRemoved() method.
> 2. revising AtomicHashMapProxy.assertValid() method to check only if the map is null (ie. removed) but no longer use the removed flag.
> 3. revising ReadCommittedEntry.commit() method to no longer call markRemoved() method.
> The consequences of these changes are:
> 1. Any further access to a stale map results in IllegalStateException ONLY in the thread that performed the removal. This thread 'knows' the map is stale so it is fine to punish it. Other threads remain unaffected until lock acquisition or commit is performed (depending on locking model).
> 2. Other threads can continue to use the previously obtained map handle for reads without danger of getting an exception.
> 3. If a write operation is done on the map, the results depend on the locking model:
> 3.1 optimistic locking + write skew check: a WriteSkewException will stop the commit during prepare
> 3.2 optimistic locking, no write skew check: the write is committed and the work of the transaction that removed the map is overwritten. The map is effectively revived.
> 3.3 pessimistic locking: same as 3.2
> Please note 3.2 and 3.3 work the same as for normal values (not atomic maps).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months
[JBoss JIRA] (ISPN-2103) Concurrent access after removal of an AtomicMap should NOT result in an IllegalStateException when accessed by other threads
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-2103?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-2103:
-----------------------------------------------
Brad Maxwell <bmaxwell(a)redhat.com> changed the Status of [bug 922699|https://bugzilla.redhat.com/show_bug.cgi?id=922699] from ASSIGNED to NEW
> Concurrent access after removal of an AtomicMap should NOT result in an IllegalStateException when accessed by other threads
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-2103
> URL: https://issues.jboss.org/browse/ISPN-2103
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.1.5.FINAL
> Reporter: Adrian Nistor
> Assignee: Tristan Tarrant
>
> ISPN-1121 introduces an IllegalStateException that is being thrown from AtomicMap methods once the map handle has become stale (ie. removed from cache). In case of concurrent access, the exception is thrown to all threads not just to the thread that performed the removal. This was fine-ish in older versions of Infinispan before introduction of optimistic and pessimistic locking but it should be reconsidered now because:
> 1. It interferes/overlaps with transaction isolation. We should rely on the selected locking scheme (OL/PL) to detect conflicts between transactions and report the problem accordingly. Especially if the stale map is used just for reading - this should be allowed to work rather than stop it.
> 2. This exception is pretty disruptive and awkward to handle. All methods of an AtomicMap can result in this exception and the current thread has to be prepared for handling it if other threads remove the map. Not much transaction isolation.
> 3. Since the TreeCache is backed by AtomicMap nearly all Tree API can throw this.
> The proposed fix consists of:
> 1. removing AtomicHashMap.removed flag and AtomicHashMap.markRemoved() method.
> 2. revising AtomicHashMapProxy.assertValid() method to check only if the map is null (ie. removed) but no longer use the removed flag.
> 3. revising ReadCommittedEntry.commit() method to no longer call markRemoved() method.
> The consequences of these changes are:
> 1. Any further access to a stale map results in IllegalStateException ONLY in the thread that performed the removal. This thread 'knows' the map is stale so it is fine to punish it. Other threads remain unaffected until lock acquisition or commit is performed (depending on locking model).
> 2. Other threads can continue to use the previously obtained map handle for reads without danger of getting an exception.
> 3. If a write operation is done on the map, the results depend on the locking model:
> 3.1 optimistic locking + write skew check: a WriteSkewException will stop the commit during prepare
> 3.2 optimistic locking, no write skew check: the write is committed and the work of the transaction that removed the map is overwritten. The map is effectively revived.
> 3.3 pessimistic locking: same as 3.2
> Please note 3.2 and 3.3 work the same as for normal values (not atomic maps).
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months
[JBoss JIRA] (ISPN-7008) IllegalLifeCycleStateException from BlockingRunnable.isReady() escapes to originator
by Dan Berindei (JIRA)
Dan Berindei created ISPN-7008:
----------------------------------
Summary: IllegalLifeCycleStateException from BlockingRunnable.isReady() escapes to originator
Key: ISPN-7008
URL: https://issues.jboss.org/browse/ISPN-7008
Project: Infinispan
Issue Type: Bug
Components: Core
Affects Versions: 8.2.4.Final, 9.0.0.Alpha4
Reporter: Dan Berindei
Fix For: 9.0.0.Beta1
The ISPN-5539 fix caught {{IllegalLifecycleStateException}} in {{BlockingRunnable.run()}} and sent a {{CacheNotFoundResponse}} to the originator instead. However, exceptions thrown from {{BlockingRunnable.isReady()}} are not caught, and are instead sent back to the originator. An example stack trace is available in the WFLY-7029 description.
In this case, the exception was thrown directly when the inbound invocation handler tried to submit the command to the remote commands executor. But I see a related problem in {{BlockingTaskAwareExecutorServiceImpl.ControllerThread.run()}}, which stops with the first exception from {{isReady()}}. For {{IllegalLifecycleStateException}} it's probably ok, but for other exceptions it should continue processing new tasks.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months
[JBoss JIRA] (ISPN-7007) Refactor the AS modules into one package
by Tristan Tarrant (JIRA)
Tristan Tarrant created ISPN-7007:
-------------------------------------
Summary: Refactor the AS modules into one package
Key: ISPN-7007
URL: https://issues.jboss.org/browse/ISPN-7007
Project: Infinispan
Issue Type: Task
Components: WildFly modules
Reporter: Tristan Tarrant
Fix For: 9.0.0.Final
- remove as-modules/client
- move as-modules/embedded to become as-modules
- for symmetry with the uberjars, have an org.infinispan.embedded and org.infinispan.remote modules which re-export the appropriate APIs.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months
[JBoss JIRA] (ISPN-7006) Module sets infinispan-as-embedded-modules and infinispan-as-client-modules are not compatible
by Sanne Grinovero (JIRA)
Sanne Grinovero created ISPN-7006:
-------------------------------------
Summary: Module sets infinispan-as-embedded-modules and infinispan-as-client-modules are not compatible
Key: ISPN-7006
URL: https://issues.jboss.org/browse/ISPN-7006
Project: Infinispan
Issue Type: Bug
Components: WildFly modules
Affects Versions: 8.2.4.Final
Reporter: Sanne Grinovero
If one downloads both packages {{infinispan-as-client-modules}} and {{infinispan-as-embedded-modules}} and unzips them both on the same appserver, Infinispan fails to initialize because of LinkageError exceptions.
The {{infinispan-as-embedded-modules}} seems to include all modules of the remote client as well, however the two structures are not fully consistent.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months
[JBoss JIRA] (ISPN-5193) infinispan-embedded and infinispan-cli-interpreter don't work together
by Sebastian Łaskawiec (JIRA)
[ https://issues.jboss.org/browse/ISPN-5193?page=com.atlassian.jira.plugin.... ]
Sebastian Łaskawiec updated ISPN-5193:
--------------------------------------
Priority: Blocker (was: Major)
> infinispan-embedded and infinispan-cli-interpreter don't work together
> ----------------------------------------------------------------------
>
> Key: ISPN-5193
> URL: https://issues.jboss.org/browse/ISPN-5193
> Project: Infinispan
> Issue Type: Bug
> Components: CLI
> Affects Versions: 7.1.0.CR2, 8.0.0.Beta2
> Reporter: Jakub Markos
> Assignee: Sebastian Łaskawiec
> Priority: Blocker
>
> Creating a java application (no container) with both infinispan-embedded and infinispan-cli-interpreter dependencies results in this error when starting a cache manager:
> {code}Exception in thread "main" java.util.ServiceConfigurationError: org.infinispan.lifecycle.ModuleLifecycle: Provider org.infinispan.cli.interpreter.LifecycleCallbacks could not be instantiated: java.lang.ExceptionInInitializerError
> at java.util.ServiceLoader.fail(ServiceLoader.java:224)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
> at org.infinispan.commons.util.ServiceFinder.addServices(ServiceFinder.java:60)
> at org.infinispan.commons.util.ServiceFinder.load(ServiceFinder.java:42)
> at org.infinispan.util.ModuleProperties.resolveModuleLifecycles(ModuleProperties.java:41)
> at org.infinispan.factories.GlobalComponentRegistry.<init>(GlobalComponentRegistry.java:94)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:292)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:271)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:244)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:231)
> at Main64.main(Main64.java:17)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
> Caused by: java.lang.ExceptionInInitializerError
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at java.lang.Class.newInstance(Class.java:374)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
> ... 15 more
> Caused by: java.lang.IllegalArgumentException: Logger implementation class org.infinispan.cli.interpreter.logging.Log_$logger has no matching constructor
> at infinispan.org.jboss.logging.Logger.getMessageLogger(Logger.java:2255)
> at infinispan.org.jboss.logging.Logger.getMessageLogger(Logger.java:2211)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:21)
> at org.infinispan.cli.interpreter.LifecycleCallbacks.<clinit>(LifecycleCallbacks.java:20)
> ... 21 more
> {code}
> Tried with 7.1.0.CR2, config:
> {code}
> <?xml version="1.0" encoding="UTF-8"?>
> <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns="urn:infinispan:config:7.1">
> <cache-container default-cache="localcache">
> <local-cache name="localcache"/>
> </cache-container>
> </infinispan>
> {code}
> application:
> {code}
> public static void main(String[] args) throws Exception {
> EmbeddedCacheManager manager = new DefaultCacheManager("config.xml");
> Cache defaultCache = manager.getCache("localcache");
> for (int i = 0; i < 10; i++) {
> defaultCache.put("key"+i, "value"+i);
> }
> Thread.sleep(5000000);
> }
> {code}
> Using infinispan-core dependency instead of infinispan-embedded works.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months
[JBoss JIRA] (ISPN-5193) infinispan-embedded and infinispan-cli-interpreter don't work together
by Sebastian Łaskawiec (JIRA)
[ https://issues.jboss.org/browse/ISPN-5193?page=com.atlassian.jira.plugin.... ]
Sebastian Łaskawiec commented on ISPN-5193:
-------------------------------------------
Unfortunately there's no perfect solution in this scenario...
The main problem here are duplicated classes (class X is available in the Uber Jar and in the module or other Jar). Even if we decided not to relocate JBoss Logging we would probably get a {{ClassCastException}} at some point (because the root problem remains the same). The situation gets even worse when you consider CDI as the spec clearly says that in case of duplicated classes, non portable behavior might occur.
You are also right about the workaround. The suggested solution is to use small jars instead.
I will look into this but I'm afraid there's not much we can do (apart from telling our users that they should not mix Uber Jars with Small Jars).
> infinispan-embedded and infinispan-cli-interpreter don't work together
> ----------------------------------------------------------------------
>
> Key: ISPN-5193
> URL: https://issues.jboss.org/browse/ISPN-5193
> Project: Infinispan
> Issue Type: Bug
> Components: CLI
> Affects Versions: 7.1.0.CR2, 8.0.0.Beta2
> Reporter: Jakub Markos
> Assignee: Sebastian Łaskawiec
>
> Creating a java application (no container) with both infinispan-embedded and infinispan-cli-interpreter dependencies results in this error when starting a cache manager:
> {code}Exception in thread "main" java.util.ServiceConfigurationError: org.infinispan.lifecycle.ModuleLifecycle: Provider org.infinispan.cli.interpreter.LifecycleCallbacks could not be instantiated: java.lang.ExceptionInInitializerError
> at java.util.ServiceLoader.fail(ServiceLoader.java:224)
> at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
> at org.infinispan.commons.util.ServiceFinder.addServices(ServiceFinder.java:60)
> at org.infinispan.commons.util.ServiceFinder.load(ServiceFinder.java:42)
> at org.infinispan.util.ModuleProperties.resolveModuleLifecycles(ModuleProperties.java:41)
> at org.infinispan.factories.GlobalComponentRegistry.<init>(GlobalComponentRegistry.java:94)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:292)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:271)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:244)
> at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:231)
> at Main64.main(Main64.java:17)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
> Caused by: java.lang.ExceptionInInitializerError
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at java.lang.Class.newInstance(Class.java:374)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
> ... 15 more
> Caused by: java.lang.IllegalArgumentException: Logger implementation class org.infinispan.cli.interpreter.logging.Log_$logger has no matching constructor
> at infinispan.org.jboss.logging.Logger.getMessageLogger(Logger.java:2255)
> at infinispan.org.jboss.logging.Logger.getMessageLogger(Logger.java:2211)
> at org.infinispan.util.logging.LogFactory.getLog(LogFactory.java:21)
> at org.infinispan.cli.interpreter.LifecycleCallbacks.<clinit>(LifecycleCallbacks.java:20)
> ... 21 more
> {code}
> Tried with 7.1.0.CR2, config:
> {code}
> <?xml version="1.0" encoding="UTF-8"?>
> <infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns="urn:infinispan:config:7.1">
> <cache-container default-cache="localcache">
> <local-cache name="localcache"/>
> </cache-container>
> </infinispan>
> {code}
> application:
> {code}
> public static void main(String[] args) throws Exception {
> EmbeddedCacheManager manager = new DefaultCacheManager("config.xml");
> Cache defaultCache = manager.getCache("localcache");
> for (int i = 0; i < 10; i++) {
> defaultCache.put("key"+i, "value"+i);
> }
> Thread.sleep(5000000);
> }
> {code}
> Using infinispan-core dependency instead of infinispan-embedded works.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 7 months