[infinispan-issues] [JBoss JIRA] (ISPN-10206) Cache fails to stop due to unreleased persistence manager semaphore permits when using early terminal stream operations
Will Burns (Jira)
issues at jboss.org
Mon May 20 12:34:00 EDT 2019
[ https://issues.jboss.org/browse/ISPN-10206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Will Burns updated ISPN-10206:
------------------------------
Description:
Wildfly 15.0.1 fails to stop on shutdown command due to Infinispan subsystem failing to stop it's persisted caches successfully due to a possible bug in Infinispan's PersistenceManagerImpl.publishEntries not releasing Semaphore permit after finishing, so that the stop method waits forever because it lacks one more permit.
To clarify this issue only happens when invoking an operation that uses an early terminating stream operation that doesn't fully iterate over the store contents. Examples would be `Cache.isEmpty`
!third_semaphore.png|thumbnail!
!forth_semaphore.png|thumbnail!
Use following application to reproduce the issue:
[^web-java-wildfly.zip]
standalone Infinispan caches:
{code}
<cache-container name="testMetaData">
<local-cache name="testCache">
<locking concurrency-level="3"/>
<transaction locking="OPTIMISTIC" mode="FULL_XA"/>
<object-memory size="1000000"/>
<file-store path="testCH/testCache" passivation="false" preload="false" purge="false"/>
</local-cache>
<local-cache name="testComponentCache">
<locking concurrency-level="3"/>
<transaction locking="OPTIMISTIC" mode="FULL_XA"/>
<object-memory size="1000000"/>
<file-store path="testCH/testComponentCache" passivation="false" preload="false" purge="false"/>
</local-cache>
</cache-container>
{code}
{code}
11:02:00,103 INFO [stdout] (default task-1) bla blanull
11:02:00,103 INFO [stdout] (default task-1) bla blanull
11:03:17,748 INFO [org.jboss.as.server] (management-handler-thread - 1) WFLYSRV0236: Suspending server with no timeout.
11:03:17,750 INFO [org.jboss.as.ejb3] (management-handler-thread - 1) WFLYEJB0493: EJB subsystem suspension complete
11:03:17,752 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
11:03:17,785 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb1
11:03:17,785 INFO [org.jboss.as.mail.extension] (MSC service thread 1-7) WFLYMAIL0002: Unbound mail session [java:/mail/myproject]
11:03:17,785 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb2
11:03:17,787 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb3
11:03:17,789 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 88) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
11:03:17,790 INFO [org.jboss.as.mail.extension] (MSC service thread 1-3) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
11:03:17,790 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb4
11:03:17,797 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) WFLYJCA0019: Stopped Driver service with driver-name = ojdbc.jar
11:03:17,800 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 101) WFLYUT0022: Unregistered web context: '/wildfly-helloworld' from server 'default-server'
11:03:17,821 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 101) MODCLUSTER000002: Initiating mod_cluster shutdown
11:03:17,823 INFO [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0019: Host default-host stopping
11:03:17,840 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-3) WFLYJCA0119: Unbinding connection factory named java:/JmsXA to alias java:jboss/DefaultJMSConnectionFactory
11:03:17,841 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
11:03:17,842 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
11:03:17,842 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = h2
11:03:17,846 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 101) WFLYCLINF0003: Stopped client-mappings cache from ejb container
11:03:17,864 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0008: Undertow HTTP listener default suspending
11:03:17,864 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0008: Undertow AJP listener ajp suspending
11:03:17,870 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0028: Stopped deployment ojdbc.jar (runtime-name: ojdbc.jar) in 108ms
11:03:17,883 INFO [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0028: Stopped deployment wildfly-helloworld.war (runtime-name: wildfly-helloworld.war) in 120ms
11:03:17,875 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to 192.168.1.2:8009
11:03:17,874 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 192.168.1.2:8080
11:03:17,872 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 104) AMQ151003: resource adaptor stopped
11:03:17,932 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 104) AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.6.3.jbossorg-00014 [7b985d04-761e-11e9-bb4e-0a0027000008] stopped, uptime 1 minute
11:03:17,933 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0008: Undertow HTTPS listener default-ssl suspending
11:03:17,934 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0007: Undertow HTTPS listener default-ssl stopped, was bound to 192.168.1.2:8443
11:03:17,935 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0004: Undertow 2.0.15.Final stopping
{code}
was:
Wildfly 15.0.1 fails to stop on shutdown command due to Infinispan subsystem failing to stop it's persisted caches successfully due to a possible bug in Infinispan's PersistenceManagerImpl.publishEntries not releasing Semaphore permit after finishing, so that the stop method waits forever because it lacks one more permit.
!third_semaphore.png|thumbnail!
!forth_semaphore.png|thumbnail!
Use following application to reproduce the issue:
[^web-java-wildfly.zip]
standalone Infinispan caches:
{code}
<cache-container name="testMetaData">
<local-cache name="testCache">
<locking concurrency-level="3"/>
<transaction locking="OPTIMISTIC" mode="FULL_XA"/>
<object-memory size="1000000"/>
<file-store path="testCH/testCache" passivation="false" preload="false" purge="false"/>
</local-cache>
<local-cache name="testComponentCache">
<locking concurrency-level="3"/>
<transaction locking="OPTIMISTIC" mode="FULL_XA"/>
<object-memory size="1000000"/>
<file-store path="testCH/testComponentCache" passivation="false" preload="false" purge="false"/>
</local-cache>
</cache-container>
{code}
{code}
11:02:00,103 INFO [stdout] (default task-1) bla blanull
11:02:00,103 INFO [stdout] (default task-1) bla blanull
11:03:17,748 INFO [org.jboss.as.server] (management-handler-thread - 1) WFLYSRV0236: Suspending server with no timeout.
11:03:17,750 INFO [org.jboss.as.ejb3] (management-handler-thread - 1) WFLYEJB0493: EJB subsystem suspension complete
11:03:17,752 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
11:03:17,785 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb1
11:03:17,785 INFO [org.jboss.as.mail.extension] (MSC service thread 1-7) WFLYMAIL0002: Unbound mail session [java:/mail/myproject]
11:03:17,785 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb2
11:03:17,787 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb3
11:03:17,789 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 88) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
11:03:17,790 INFO [org.jboss.as.mail.extension] (MSC service thread 1-3) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
11:03:17,790 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb4
11:03:17,797 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) WFLYJCA0019: Stopped Driver service with driver-name = ojdbc.jar
11:03:17,800 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 101) WFLYUT0022: Unregistered web context: '/wildfly-helloworld' from server 'default-server'
11:03:17,821 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 101) MODCLUSTER000002: Initiating mod_cluster shutdown
11:03:17,823 INFO [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0019: Host default-host stopping
11:03:17,840 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-3) WFLYJCA0119: Unbinding connection factory named java:/JmsXA to alias java:jboss/DefaultJMSConnectionFactory
11:03:17,841 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
11:03:17,842 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
11:03:17,842 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = h2
11:03:17,846 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 101) WFLYCLINF0003: Stopped client-mappings cache from ejb container
11:03:17,864 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0008: Undertow HTTP listener default suspending
11:03:17,864 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0008: Undertow AJP listener ajp suspending
11:03:17,870 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0028: Stopped deployment ojdbc.jar (runtime-name: ojdbc.jar) in 108ms
11:03:17,883 INFO [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0028: Stopped deployment wildfly-helloworld.war (runtime-name: wildfly-helloworld.war) in 120ms
11:03:17,875 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to 192.168.1.2:8009
11:03:17,874 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 192.168.1.2:8080
11:03:17,872 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 104) AMQ151003: resource adaptor stopped
11:03:17,932 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 104) AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.6.3.jbossorg-00014 [7b985d04-761e-11e9-bb4e-0a0027000008] stopped, uptime 1 minute
11:03:17,933 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0008: Undertow HTTPS listener default-ssl suspending
11:03:17,934 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0007: Undertow HTTPS listener default-ssl stopped, was bound to 192.168.1.2:8443
11:03:17,935 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0004: Undertow 2.0.15.Final stopping
{code}
> Cache fails to stop due to unreleased persistence manager semaphore permits when using early terminal stream operations
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-10206
> URL: https://issues.jboss.org/browse/ISPN-10206
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Loaders and Stores
> Affects Versions: 10.0.0.Beta3, 9.4.13.Final
> Environment: Windows 10 x64
> java version "1.8.0_212"
> Java(TM) SE Runtime Environment (build 1.8.0_212-b10)
> Java HotSpot(TM) 64-Bit Server VM (build 25.212-b10, mixed mode)
> Wildfly 15.0.1
> Reporter: Paul Ferraro
> Assignee: Will Burns
> Priority: Major
> Fix For: 10.0.0.Beta4, 9.4.14.Final
>
>
> Wildfly 15.0.1 fails to stop on shutdown command due to Infinispan subsystem failing to stop it's persisted caches successfully due to a possible bug in Infinispan's PersistenceManagerImpl.publishEntries not releasing Semaphore permit after finishing, so that the stop method waits forever because it lacks one more permit.
> To clarify this issue only happens when invoking an operation that uses an early terminating stream operation that doesn't fully iterate over the store contents. Examples would be `Cache.isEmpty`
> !third_semaphore.png|thumbnail!
> !forth_semaphore.png|thumbnail!
> Use following application to reproduce the issue:
> [^web-java-wildfly.zip]
> standalone Infinispan caches:
> {code}
> <cache-container name="testMetaData">
> <local-cache name="testCache">
> <locking concurrency-level="3"/>
> <transaction locking="OPTIMISTIC" mode="FULL_XA"/>
> <object-memory size="1000000"/>
> <file-store path="testCH/testCache" passivation="false" preload="false" purge="false"/>
> </local-cache>
> <local-cache name="testComponentCache">
> <locking concurrency-level="3"/>
> <transaction locking="OPTIMISTIC" mode="FULL_XA"/>
> <object-memory size="1000000"/>
> <file-store path="testCH/testComponentCache" passivation="false" preload="false" purge="false"/>
> </local-cache>
> </cache-container>
> {code}
> {code}
> 11:02:00,103 INFO [stdout] (default task-1) bla blanull
> 11:02:00,103 INFO [stdout] (default task-1) bla blanull
> 11:03:17,748 INFO [org.jboss.as.server] (management-handler-thread - 1) WFLYSRV0236: Suspending server with no timeout.
> 11:03:17,750 INFO [org.jboss.as.ejb3] (management-handler-thread - 1) WFLYEJB0493: EJB subsystem suspension complete
> 11:03:17,752 INFO [org.jboss.as.server] (Management Triggered Shutdown) WFLYSRV0241: Shutting down in response to management operation 'shutdown'
> 11:03:17,785 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-4) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb1
> 11:03:17,785 INFO [org.jboss.as.mail.extension] (MSC service thread 1-7) WFLYMAIL0002: Unbound mail session [java:/mail/myproject]
> 11:03:17,785 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-3) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb2
> 11:03:17,787 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb3
> 11:03:17,789 INFO [org.wildfly.extension.messaging-activemq] (ServerService Thread Pool -- 88) WFLYMSGAMQ0006: Unbound messaging object to jndi name java:/ConnectionFactory
> 11:03:17,790 INFO [org.jboss.as.mail.extension] (MSC service thread 1-3) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
> 11:03:17,790 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-8) WFLYJCA0099: Unbound non-transactional data source: java:jboss/datasources/myprojdb4
> 11:03:17,797 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-4) WFLYJCA0019: Stopped Driver service with driver-name = ojdbc.jar
> 11:03:17,800 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 101) WFLYUT0022: Unregistered web context: '/wildfly-helloworld' from server 'default-server'
> 11:03:17,821 INFO [org.jboss.modcluster] (ServerService Thread Pool -- 101) MODCLUSTER000002: Initiating mod_cluster shutdown
> 11:03:17,823 INFO [org.wildfly.extension.undertow] (MSC service thread 1-8) WFLYUT0019: Host default-host stopping
> 11:03:17,840 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-3) WFLYJCA0119: Unbinding connection factory named java:/JmsXA to alias java:jboss/DefaultJMSConnectionFactory
> 11:03:17,841 INFO [org.jboss.as.connector.deployment] (MSC service thread 1-4) WFLYJCA0011: Unbound JCA ConnectionFactory [java:/JmsXA]
> 11:03:17,842 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> 11:03:17,842 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = h2
> 11:03:17,846 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 101) WFLYCLINF0003: Stopped client-mappings cache from ejb container
> 11:03:17,864 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0008: Undertow HTTP listener default suspending
> 11:03:17,864 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0008: Undertow AJP listener ajp suspending
> 11:03:17,870 INFO [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0028: Stopped deployment ojdbc.jar (runtime-name: ojdbc.jar) in 108ms
> 11:03:17,883 INFO [org.jboss.as.server.deployment] (MSC service thread 1-8) WFLYSRV0028: Stopped deployment wildfly-helloworld.war (runtime-name: wildfly-helloworld.war) in 120ms
> 11:03:17,875 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0007: Undertow AJP listener ajp stopped, was bound to 192.168.1.2:8009
> 11:03:17,874 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 192.168.1.2:8080
> 11:03:17,872 INFO [org.apache.activemq.artemis.ra] (ServerService Thread Pool -- 104) AMQ151003: resource adaptor stopped
> 11:03:17,932 INFO [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 104) AMQ221002: Apache ActiveMQ Artemis Message Broker version 2.6.3.jbossorg-00014 [7b985d04-761e-11e9-bb4e-0a0027000008] stopped, uptime 1 minute
> 11:03:17,933 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0008: Undertow HTTPS listener default-ssl suspending
> 11:03:17,934 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0007: Undertow HTTPS listener default-ssl stopped, was bound to 192.168.1.2:8443
> 11:03:17,935 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0004: Undertow 2.0.15.Final stopping
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
More information about the infinispan-issues
mailing list