[Red Hat JIRA] (ISPN-12570) REST server stop hangs occasionally
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-12570?page=com.atlassian.jira.plugi... ]
Dan Berindei updated ISPN-12570:
--------------------------------
Status: Open (was: New)
> REST server stop hangs occasionally
> -----------------------------------
>
> Key: ISPN-12570
> URL: https://issues.redhat.com/browse/ISPN-12570
> Project: Infinispan
> Issue Type: Bug
> Components: REST, Server
> Affects Versions: 12.0.0.Dev07
> Reporter: Dan Berindei
> Priority: Major
> Fix For: 12.0.0.CR1
>
>
> {{NettyTransport.stop()}} calls {{acceptedChannels.close()}} after shutting down the both the {{masterGroup}} and the {{ioGroup}}. But if a channel is still open, closing it requires submitting a task to the channel's event loop, which is now shut down.
> {{AbstractChannelHandlerContext.safeExecute()}} hides the rejection exception, but the {{ChannelPromise}} returned by {{channel.close()}} never completes, and the server doesn't stop.
> {noformat}
> java.util.concurrent.RejectedExecutionException: event executor terminated
> at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926)
> at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353)
> at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346)
> at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828)
> at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818)
> at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:989)
> at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608)
> at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:472)
> at io.netty.channel.DefaultChannelPipeline.close(DefaultChannelPipeline.java:957)
> at io.netty.channel.AbstractChannel.close(AbstractChannel.java:232)
> at io.netty.channel.group.DefaultChannelGroup.close(DefaultChannelGroup.java:342)
> at io.netty.channel.group.DefaultChannelGroup.close(DefaultChannelGroup.java:221)
> at org.infinispan.server.core.transport.NettyTransport.stop(NettyTransport.java:135)
> at org.infinispan.server.core.AbstractProtocolServer.stop(AbstractProtocolServer.java:202)
> at org.infinispan.rest.RestServer.stop(RestServer.java:100)
> at org.infinispan.rest.helper.RestServerHelper.stop(RestServerHelper.java:86)
> {noformat}
> {noformat}
> at java.lang.Object.$$BlockHound$$_wait(Object.java:-2)
> at java.lang.Object.wait(Object.java:-1)
> at java.lang.Object.wait(Object.java:328)
> at io.netty.util.concurrent.DefaultPromise.awaitUninterruptibly(DefaultPromise.java:264)
> at io.netty.channel.group.DefaultChannelGroupFuture.awaitUninterruptibly(DefaultChannelGroupFuture.java:178)
> at io.netty.channel.group.DefaultChannelGroupFuture.awaitUninterruptibly(DefaultChannelGroupFuture.java:41)
> at org.infinispan.server.core.transport.NettyTransport.stop(NettyTransport.java:149)
> at org.infinispan.server.core.AbstractProtocolServer.stop(AbstractProtocolServer.java:202)
> at org.infinispan.rest.RestServer.stop(RestServer.java:100)
> at org.infinispan.rest.helper.RestServerHelper.stop(RestServerHelper.java:86)
> {noformat}
> {{CacheV2ResourceTest.afterSuite()}} sometimes hangs this way. Initially I attributed the hanging to ISPN-12558, but it looks like a separate issue.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 6 months
[Red Hat JIRA] (ISPN-10236) Hotrod client error releasing channel after server cache stop
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-10236?page=com.atlassian.jira.plugi... ]
Dan Berindei edited comment on ISPN-10236 at 12/11/20 1:53 AM:
---------------------------------------------------------------
Fixed with ISPN-11885.
I'm ignoring the error in the latest comment, because there the entire server is stopping.
was (Author: dan.berindei):
Fixed with ISPN-11885.
I'm ignoring the error in the latest comment, because it's a server problem.
> Hotrod client error releasing channel after server cache stop
> -------------------------------------------------------------
>
> Key: ISPN-10236
> URL: https://issues.redhat.com/browse/ISPN-10236
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite
> Affects Versions: 10.0.0.Beta3
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Attachments: ISPN-10137_package_private_scope_20190524-1732_ServerFailureRetryTest-infinispan-client-hotrod.log.gz
>
>
> Random failure in {{ServerFailureRetryTest.testRetryCacheStopped}} caused by an assert statement in {{ChannelPool.release()}}.
> {noformat}
> 17:37:36,562 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.client.hotrod.retry.ServerFailureRetryTest.testRetryCacheStopped
> java.lang.AssertionError: Error releasing [id: 0x5d9755e6, L:/127.0.0.1:42472 ! R:127.0.0.1/127.0.0.1:44865]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelPool.release(ChannelPool.java:170) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelFactory.releaseChannel(ChannelFactory.java:309) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.operations.HotRodOperation.releaseChannel(HotRodOperation.java:105) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.invoke(RetryOnFailureOperation.java:80) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelPool.activateChannel(ChannelPool.java:217) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelPool.acquire(ChannelPool.java:86) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelFactory.fetchChannelAndInvoke(ChannelFactory.java:259) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelFactory.fetchChannelAndInvoke(ChannelFactory.java:297) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.fetchChannelAndInvoke(AbstractKeyOperation.java:41) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:61) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putAsync(RemoteCacheImpl.java:366) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:334) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79) ~[classes/:?]
> at org.infinispan.client.hotrod.retry.ServerFailureRetryTest.testRetryCacheStopped(ServerFailureRetryTest.java:63) ~[test-classes/:?]
> {noformat}
> I investigated a bit and I couldn't find an obvious mistake in the way {{ChannelPool.created}} is incremented and decremented, but I think it would help if access to it and {{ChannelPool.active}} were centralized in a smaller number of methods.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 6 months
[Red Hat JIRA] (ISPN-10236) Hotrod client error releasing channel after server cache stop
by Dan Berindei (Jira)
[ https://issues.redhat.com/browse/ISPN-10236?page=com.atlassian.jira.plugi... ]
Dan Berindei resolved ISPN-10236.
---------------------------------
Resolution: Done
Fixed with ISPN-11885.
I'm ignoring the error in the latest comment, because it's a server problem.
> Hotrod client error releasing channel after server cache stop
> -------------------------------------------------------------
>
> Key: ISPN-10236
> URL: https://issues.redhat.com/browse/ISPN-10236
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite
> Affects Versions: 10.0.0.Beta3
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Attachments: ISPN-10137_package_private_scope_20190524-1732_ServerFailureRetryTest-infinispan-client-hotrod.log.gz
>
>
> Random failure in {{ServerFailureRetryTest.testRetryCacheStopped}} caused by an assert statement in {{ChannelPool.release()}}.
> {noformat}
> 17:37:36,562 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.client.hotrod.retry.ServerFailureRetryTest.testRetryCacheStopped
> java.lang.AssertionError: Error releasing [id: 0x5d9755e6, L:/127.0.0.1:42472 ! R:127.0.0.1/127.0.0.1:44865]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelPool.release(ChannelPool.java:170) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelFactory.releaseChannel(ChannelFactory.java:309) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.operations.HotRodOperation.releaseChannel(HotRodOperation.java:105) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.invoke(RetryOnFailureOperation.java:80) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelPool.activateChannel(ChannelPool.java:217) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelPool.acquire(ChannelPool.java:86) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelFactory.fetchChannelAndInvoke(ChannelFactory.java:259) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.transport.netty.ChannelFactory.fetchChannelAndInvoke(ChannelFactory.java:297) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.operations.AbstractKeyOperation.fetchChannelAndInvoke(AbstractKeyOperation.java:41) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:61) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putAsync(RemoteCacheImpl.java:366) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.put(RemoteCacheImpl.java:334) ~[classes/:?]
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.put(RemoteCacheSupport.java:79) ~[classes/:?]
> at org.infinispan.client.hotrod.retry.ServerFailureRetryTest.testRetryCacheStopped(ServerFailureRetryTest.java:63) ~[test-classes/:?]
> {noformat}
> I investigated a bit and I couldn't find an obvious mistake in the way {{ChannelPool.created}} is incremented and decremented, but I think it would help if access to it and {{ChannelPool.active}} were centralized in a smaller number of methods.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 6 months
[Red Hat JIRA] (ISPN-12569) SearchWorkspaceImpl is blocking
by Will Burns (Jira)
Will Burns created ISPN-12569:
---------------------------------
Summary: SearchWorkspaceImpl is blocking
Key: ISPN-12569
URL: https://issues.redhat.com/browse/ISPN-12569
Project: Infinispan
Issue Type: Bug
Components: Embedded Querying
Reporter: Will Burns
The SearchWorkspaceImpl has two purge methods that both block waiting for the stage returned from hibernate internal clases. We should change this API to support non blocking.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 6 months
[Red Hat JIRA] (ISPN-12568) Docs: Add help for alias command
by Donald Naro (Jira)
[ https://issues.redhat.com/browse/ISPN-12568?page=com.atlassian.jira.plugi... ]
Donald Naro updated ISPN-12568:
-------------------------------
Sprint: DataGrid Sprint #53
> Docs: Add help for alias command
> --------------------------------
>
> Key: ISPN-12568
> URL: https://issues.redhat.com/browse/ISPN-12568
> Project: Infinispan
> Issue Type: Enhancement
> Components: Documentation
> Reporter: Donald Naro
> Assignee: Donald Naro
> Priority: Major
> Fix For: 12.0.0.Final, 11.0.9.Final
>
>
> Need help for creating command aliases.
>
> [disconnected]>
> alias clear clusters connect delete encoding help migrate quit shell uninstall version
> benchmark cluster config credentials echo export install patch run unalias user
> [disconnected]> help alias
> [disconnected]>
>
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 6 months
[Red Hat JIRA] (ISPN-12568) Docs: Add help for alias command
by Donald Naro (Jira)
[ https://issues.redhat.com/browse/ISPN-12568?page=com.atlassian.jira.plugi... ]
Donald Naro updated ISPN-12568:
-------------------------------
Status: Open (was: New)
> Docs: Add help for alias command
> --------------------------------
>
> Key: ISPN-12568
> URL: https://issues.redhat.com/browse/ISPN-12568
> Project: Infinispan
> Issue Type: Enhancement
> Components: Documentation
> Reporter: Donald Naro
> Assignee: Donald Naro
> Priority: Major
> Fix For: 12.0.0.Final, 11.0.9.Final
>
>
> Need help for creating command aliases.
>
> [disconnected]>
> alias clear clusters connect delete encoding help migrate quit shell uninstall version
> benchmark cluster config credentials echo export install patch run unalias user
> [disconnected]> help alias
> [disconnected]>
>
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
3 years, 6 months