[JBoss JIRA] (ISPN-11836) MassIndexLock should be non blocking
by Will Burns (Jira)
Will Burns created ISPN-11836:
---------------------------------
Summary: MassIndexLock should be non blocking
Key: ISPN-11836
URL: https://issues.redhat.com/browse/ISPN-11836
Project: Infinispan
Issue Type: Bug
Reporter: Will Burns
The MassIndexLock interface defines blocking API for it. There is no reason this can't instead be non blocking and have the DistributedExecutorMassIndexer utilize that in a non blocking fashion, it already returns a CompletionStage.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (ISPN-11835) Clustered Lock creation is blocking
by Will Burns (Jira)
Will Burns created ISPN-11835:
---------------------------------
Summary: Clustered Lock creation is blocking
Key: ISPN-11835
URL: https://issues.redhat.com/browse/ISPN-11835
Project: Infinispan
Issue Type: Bug
Components: Clustered Locks
Reporter: Will Burns
Creating a new clustered lock is blocking
A sample stack trace is as follows:
{code}
<![CDATA[java.lang.AssertionError: Blocking call! jdk.internal.misc.Unsafe#park on thread Thread[REST-IndexedRestSearchTest-IndexedRestSearchTest-NodeA-ServerIO-32-1,5,main]
at org.infinispan.util.CoreTestBlockHoundIntegration.lambda$applyTo$0(CoreTestBlockHoundIntegration.java:44)
at reactor.blockhound.BlockHound$Builder.lambda$install$8(BlockHound.java:383)
at reactor.blockhound.BlockHoundRuntime.checkBlocking(BlockHoundRuntime.java:89)
at java.base/jdk.internal.misc.Unsafe.park(Unsafe.java)
at java.base/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1798)
at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3128)
at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1868)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
at org.infinispan.util.concurrent.CompletableFutures.await(CompletableFutures.java:125)
at org.infinispan.interceptors.impl.SimpleAsyncInvocationStage.get(SimpleAsyncInvocationStage.java:36)
at org.infinispan.interceptors.impl.AsyncInterceptorChainImpl.invoke(AsyncInterceptorChainImpl.java:246)
at org.infinispan.cache.impl.InvocationHelper.doInvoke(InvocationHelper.java:298)
at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:102)
at org.infinispan.cache.impl.InvocationHelper.invoke(InvocationHelper.java:84)
at org.infinispan.cache.impl.CacheImpl.putIfAbsent(CacheImpl.java:1340)
at org.infinispan.cache.impl.DecoratedCache.putIfAbsent(DecoratedCache.java:685)
at org.infinispan.cache.impl.DecoratedCache.putIfAbsent(DecoratedCache.java:590)
at org.infinispan.cache.impl.AbstractDelegatingCache.putIfAbsent(AbstractDelegatingCache.java:349)
at org.infinispan.cache.impl.AbstractDelegatingCache.putIfAbsent(AbstractDelegatingCache.java:349)
at org.infinispan.cache.impl.AbstractDelegatingCache.putIfAbsent(AbstractDelegatingCache.java:349)
at org.infinispan.cache.impl.AbstractDelegatingCache.putIfAbsent(AbstractDelegatingCache.java:349)
at org.infinispan.cache.impl.AbstractDelegatingCache.putIfAbsent(AbstractDelegatingCache.java:349)
at org.infinispan.cache.impl.EncoderCache.putIfAbsent(EncoderCache.java:491)
at org.infinispan.lock.impl.manager.EmbeddedClusteredLockManager.createLock(EmbeddedClusteredLockManager.java:129)
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
at org.infinispan.lock.impl.manager.EmbeddedClusteredLockManager.get(EmbeddedClusteredLockManager.java:120)
at org.infinispan.query.impl.massindex.DistributedMassIndexerLock.getLock(DistributedMassIndexerLock.java:59)
at org.infinispan.query.impl.massindex.DistributedMassIndexerLock.lock(DistributedMassIndexerLock.java:30)
at org.infinispan.query.impl.massindex.DistributedExecutorMassIndexer.executeInternal(DistributedExecutorMassIndexer.java:111)
at org.infinispan.query.impl.massindex.DistributedExecutorMassIndexer.purge(DistributedExecutorMassIndexer.java:74)
at org.infinispan.rest.resources.SearchAdminResource.runMassIndexer(SearchAdminResource.java:117)
at org.infinispan.rest.resources.SearchAdminResource.clearIndexes(SearchAdminResource.java:63)
at org.infinispan.rest.framework.impl.RestDispatcherImpl.dispatch(RestDispatcherImpl.java:75)
at org.infinispan.rest.RestRequestHandler.handleRestRequest(RestRequestHandler.java:124)
at org.infinispan.rest.RestRequestHandler.channelRead0(RestRequestHandler.java:81)
at org.infinispan.rest.Http11RequestHandler.channelRead0(Http11RequestHandler.java:33)
at org.infinispan.rest.Http11RequestHandler.channelRead0(Http11RequestHandler.java:16)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.http.cors.CorsHandler.channelRead(CorsHandler.java:95)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at org.infinispan.server.core.transport.StatsChannelHandler.channelRead(StatsChannelHandler.java:26)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
{code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (ISPN-11834) Rest ResponseWrite CHUNKED_FILE response is blocking
by Will Burns (Jira)
Will Burns created ISPN-11834:
---------------------------------
Summary: Rest ResponseWrite CHUNKED_FILE response is blocking
Key: ISPN-11834
URL: https://issues.redhat.com/browse/ISPN-11834
Project: Infinispan
Issue Type: Bug
Components: REST
Reporter: Will Burns
The CHUNKED_FILE annotation blocks while trying to write the response since it has to read from the underlying disk. This can be seen from the StaticResourceTest.
Here is an example stack trace:
{code}
<failure type="java.lang.AssertionError" message="Blocking call! java.io.RandomAccessFile#readBytes on thread Thread[REST-StaticResourceTest-StaticResourceTest-NodeA-ServerIO-81-1,5,main]">
<![CDATA[java.lang.AssertionError: Blocking call! java.io.RandomAccessFile#readBytes on thread Thread[REST-StaticResourceTest-StaticResourceTest-NodeA-ServerIO-81-1,5,main]
at org.infinispan.util.CoreTestBlockHoundIntegration.lambda$applyTo$0(CoreTestBlockHoundIntegration.java:44)
at reactor.blockhound.BlockHound$Builder.lambda$install$8(BlockHound.java:383)
at reactor.blockhound.BlockHoundRuntime.checkBlocking(BlockHoundRuntime.java:89)
at java.base/java.io.RandomAccessFile.readBytes(RandomAccessFile.java)
at java.base/java.io.RandomAccessFile.read(RandomAccessFile.java:406)
at java.base/java.io.RandomAccessFile.readFully(RandomAccessFile.java:470)
at io.netty.handler.stream.ChunkedFile.readChunk(ChunkedFile.java:149)
at io.netty.handler.stream.ChunkedFile.readChunk(ChunkedFile.java:35)
at io.netty.handler.codec.http.HttpChunkedInput.readChunk(HttpChunkedInput.java:102)
at io.netty.handler.codec.http.HttpChunkedInput.readChunk(HttpChunkedInput.java:43)
at io.netty.handler.stream.ChunkedWriteHandler.doFlush(ChunkedWriteHandler.java:230)
at io.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:132)
at io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
at io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)
at io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
at org.infinispan.rest.ResponseWriter$3.writeResponse(ResponseWriter.java:69)
at org.infinispan.rest.BaseHttpRequestHandler.lambda$sendResponse$0(BaseHttpRequestHandler.java:46)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
{code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (ISPN-11832) ProtobufMetadataManagerInterceptor is blocking
by Will Burns (Jira)
Will Burns created ISPN-11832:
---------------------------------
Summary: ProtobufMetadataManagerInterceptor is blocking
Key: ISPN-11832
URL: https://issues.redhat.com/browse/ISPN-11832
Project: Infinispan
Issue Type: Bug
Components: Indexing
Reporter: Will Burns
Fix For: 11.0.0.Final
The interceptor uses invoke on the AsyncInterceptorChain in multiple places. This invocation can and will block. We should be using invokeAsync and chaining the stage.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (ISPN-11831) Add blockhound to server rest module
by Will Burns (Jira)
Will Burns created ISPN-11831:
---------------------------------
Summary: Add blockhound to server rest module
Key: ISPN-11831
URL: https://issues.redhat.com/browse/ISPN-11831
Project: Infinispan
Issue Type: Enhancement
Components: REST
Reporter: Will Burns
Assignee: Will Burns
The Server rest module has many resources that must ensure they are non blocking, otherwise it will hold the netty thread for too long. We should add blockhound to find offenders.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months