[JBoss JIRA] (ISPN-5270) Deadlock in InfinispanDirectoryProvider startup
by Dan Berindei (JIRA)
Dan Berindei created ISPN-5270:
----------------------------------
Summary: Deadlock in InfinispanDirectoryProvider startup
Key: ISPN-5270
URL: https://issues.jboss.org/browse/ISPN-5270
Project: Infinispan
Issue Type: Bug
Components: Embedded Querying
Affects Versions: 7.1.1.Final, 7.2.0.Alpha1
Reporter: Dan Berindei
Priority: Critical
The InfinispanDirectoryProvider tries to start the metadata, data, and locking caches when it starts up, with {{DefaultCacheManager.startCaches()}}.
However, when one of these caches (e.g. the metadata cache) starts, the {{LifecycleManager.cacheStarting()}}, which can then try to start the InfinispanDirectoryProvider again:
{noformat}
"CacheStartThread,null,LuceneIndexesMetadata" prio=10 tid=0x00007f5f74484000 nid=0xe42 in Object.wait() [0x00007f5efff48000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c2180000> (a org.infinispan.manager.DefaultCacheManager$1)
at java.lang.Thread.join(Thread.java:1281)
- locked <0x00000000c2180000> (a org.infinispan.manager.DefaultCacheManager$1)
at java.lang.Thread.join(Thread.java:1355)
at org.infinispan.manager.DefaultCacheManager.startCaches(DefaultCacheManager.java:465)
at org.hibernate.search.infinispan.spi.InfinispanDirectoryProvider.start(InfinispanDirectoryProvider.java:84)
at org.hibernate.search.indexes.spi.DirectoryBasedIndexManager.initialize(DirectoryBasedIndexManager.java:88)
at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:256)
at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManager(IndexManagerHolder.java:513)
- locked <0x00000000ce6001d0> (a org.hibernate.search.indexes.impl.IndexManagerHolder)
at org.hibernate.search.indexes.impl.IndexManagerHolder.createIndexManagers(IndexManagerHolder.java:482)
at org.hibernate.search.indexes.impl.IndexManagerHolder.buildEntityIndexBinding(IndexManagerHolder.java:91)
- locked <0x00000000ce6001d0> (a org.hibernate.search.indexes.impl.IndexManagerHolder)
at org.hibernate.search.spi.SearchIntegratorBuilder.initDocumentBuilders(SearchIntegratorBuilder.java:366)
at org.hibernate.search.spi.SearchIntegratorBuilder.buildNewSearchFactory(SearchIntegratorBuilder.java:204)
at org.hibernate.search.spi.SearchIntegratorBuilder.buildSearchIntegrator(SearchIntegratorBuilder.java:122)
at org.hibernate.search.spi.SearchFactoryBuilder.buildSearchFactory(SearchFactoryBuilder.java:35)
at org.infinispan.query.impl.LifecycleManager.getSearchFactory(LifecycleManager.java:260)
at org.infinispan.query.impl.LifecycleManager.cacheStarting(LifecycleManager.java:102)
at org.infinispan.factories.ComponentRegistry.notifyCacheStarting(ComponentRegistry.java:230)
at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:216)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:814)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:591)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:546)
at org.infinispan.manager.DefaultCacheManager.access$100(DefaultCacheManager.java:115)
at org.infinispan.manager.DefaultCacheManager$1.run(DefaultCacheManager.java:452)
{noformat}
This can hang the test, the attached thread dumps show {{EmbeddedCompatTest}} and {{IndexCacheStopTest}}.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 1 month
[JBoss JIRA] (ISPN-5269) Process and generate component metadata for tests
by Adrian Nistor (JIRA)
Adrian Nistor created ISPN-5269:
-----------------------------------
Summary: Process and generate component metadata for tests
Key: ISPN-5269
URL: https://issues.jboss.org/browse/ISPN-5269
Project: Infinispan
Issue Type: Enhancement
Components: Build process
Affects Versions: 7.2.0.Alpha1
Reporter: Adrian Nistor
Assignee: Adrian Nistor
Fix For: 7.2.0.Beta1
Currently it is not possible to use @Inject annotations in test classes because component metadata is not processed for test sources. It would be nice to have this (optionally) generated.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 1 month
[JBoss JIRA] (ISPN-5268) Backup segments may not be removed after failed state transfer
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-5268?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-5268:
-------------------------------
Status: Open (was: New)
> Backup segments may not be removed after failed state transfer
> ---------------------------------------------------------------
>
> Key: ISPN-5268
> URL: https://issues.jboss.org/browse/ISPN-5268
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Reporter: Tristan Tarrant
> Assignee: Dan Berindei
>
> Dist cache can lost data if the primary and all backups are lost at the same time. To detect such data loss, they are monitoring the tatal sum of entries from all nodes using JMX. If the sum reduces by a considerable ammount, data loss (by lost a segment) is concluded. But they found a case that the sum increases, not decreases, after failed state transfer. This is probably caused by an extra backup segment that hasn't been cleaned up when the last state transfer failed.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 1 month
[JBoss JIRA] (ISPN-5221) Java Hotrod client, nearcache broken after a org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException
by Enrico Olivelli (JIRA)
[ https://issues.jboss.org/browse/ISPN-5221?page=com.atlassian.jira.plugin.... ]
Enrico Olivelli commented on ISPN-5221:
---------------------------------------
I think that handling network errors inside the listener thread will be enough in order to fix my scenario.
As soon as you push your changes to the public repo I will give it a try
Thank you very much
> Java Hotrod client, nearcache broken after a org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-5221
> URL: https://issues.jboss.org/browse/ISPN-5221
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.1.0.Final
> Environment: Infinispan 7.1.0Final
> Reporter: Enrico Olivelli
> Assignee: Galder Zamarreño
> Fix For: 7.2.0.Final
>
>
> Using the LAZY nearcache (new in 7.1.0Final) when a java.net.SocketTimeoutException occurs against one of the given HotRod server the RemoteCache becames not usable any more
> This happens very frequently in our DEV environment, but I cannot reproduce it inside a test case
> This is the stacktrace
> {code}
> 15/02/13 14:51:39 ERROR event.ClientListenerNotifier: ISPN004043: Unrecoverable error reading event from server xxx.xxx.xxx/10.168.10.117:11222, exiting event reader thread
> org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:184)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:282)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readEvent(Codec20.java:126)
> at org.infinispan.client.hotrod.event.ClientListenerNotifier$EventDispatcher.run(ClientListenerNotifier.java:236)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.SocketTimeoutException
> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:179)
> ... 8 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 1 month
[JBoss JIRA] (ISPN-5268) Backup segments may not be removed after failed state transfer
by Tristan Tarrant (JIRA)
Tristan Tarrant created ISPN-5268:
-------------------------------------
Summary: Backup segments may not be removed after failed state transfer
Key: ISPN-5268
URL: https://issues.jboss.org/browse/ISPN-5268
Project: Infinispan
Issue Type: Bug
Components: Core
Reporter: Tristan Tarrant
Assignee: Dan Berindei
Dist cache can lost data if the primary and all backups are lost at the same time. To detect such data loss, they are monitoring the tatal sum of entries from all nodes using JMX. If the sum reduces by a considerable ammount, data loss (by lost a segment) is concluded. But they found a case that the sum increases, not decreases, after failed state transfer. This is probably caused by an extra backup segment that hasn't been cleaned up when the last state transfer failed.
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 1 month
[JBoss JIRA] (ISPN-5221) Java Hotrod client, nearcache broken after a org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-5221?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-5221:
----------------------------------------
Actually, the marshaller did not make it a difference, it still works fine even after the socket timeout errors have been recovered from.
> Java Hotrod client, nearcache broken after a org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-5221
> URL: https://issues.jboss.org/browse/ISPN-5221
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.1.0.Final
> Environment: Infinispan 7.1.0Final
> Reporter: Enrico Olivelli
> Assignee: Galder Zamarreño
> Fix For: 7.2.0.Final
>
>
> Using the LAZY nearcache (new in 7.1.0Final) when a java.net.SocketTimeoutException occurs against one of the given HotRod server the RemoteCache becames not usable any more
> This happens very frequently in our DEV environment, but I cannot reproduce it inside a test case
> This is the stacktrace
> {code}
> 15/02/13 14:51:39 ERROR event.ClientListenerNotifier: ISPN004043: Unrecoverable error reading event from server xxx.xxx.xxx/10.168.10.117:11222, exiting event reader thread
> org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:184)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:282)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readEvent(Codec20.java:126)
> at org.infinispan.client.hotrod.event.ClientListenerNotifier$EventDispatcher.run(ClientListenerNotifier.java:236)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.net.SocketTimeoutException
> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:179)
> ... 8 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.11#6341)
11 years, 1 month