[JBoss JIRA] (ISPN-5875) HotRod Client does not handle failover silent when nodes are stopped if using role based authentication
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5875?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5875:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> HotRod Client does not handle failover silent when nodes are stopped if using role based authentication
> -------------------------------------------------------------------------------------------------------
>
> Key: ISPN-5875
> URL: https://issues.jboss.org/browse/ISPN-5875
> Project: Infinispan
> Issue Type: Bug
> Components: Remote Protocols, Security
> Reporter: Wolf-Dieter Fink
> Assignee: Galder Zamarreño
> Priority: Critical
> Labels: hotrod, hotrod-java-client, security
> Fix For: 8.1.0.Beta1, 8.1.0.Final, 8.0.2.Final
>
>
> A HotRod client fail over silent if one instance is shutting down if there is no Security enabled.
> With role based authentication the client fail sporadically with a WARN message
> {code}WARN: ISPN004005: Error received from the server: java.security.PrivilegedActionException:
> org.infinispan.IllegalLifecycleStateException: ISPN000324: Cache 'testCache' is in 'STOPPING' state and this is an invocation not
> belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate the cache container.
> {code}
> and the following Exception:
> {code}org.infinispan.client.hotrod.exceptions.HotRodClientException:Request for message id[2825] returned server error (status=0x85):
> java.security.PrivilegedActionException: org.infinispan.IllegalLifecycleStateException: ISPN000324: Cache 'testCache' is in 'STOPPING'
> state and this is an invocation not belonging to an on-going transaction, so it does not accept new invocations. Either restart it or recreate
> the cache container.
> at org.infinispan.client.hotrod.impl.protocol.Codec20.checkForErrorsInResponseStatus(Codec20.java:336)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readPartialHeader(Codec20.java:126)
> at org.infinispan.client.hotrod.impl.protocol.Codec20.readHeader(Codec20.java:112)
> at org.infinispan.client.hotrod.impl.operations.HotRodOperation.readHeaderAndValidate(HotRodOperation.java:56)
> at org.infinispan.client.hotrod.impl.operations.AbstractKeyValueOperation.sendPutOperation(AbstractKeyValueOperation.java:57)
> at org.infinispan.client.hotrod.impl.operations.PutIfAbsentOperation.executeOperation(PutIfAbsentOperation.java:36)
> at org.infinispan.client.hotrod.impl.operations.PutIfAbsentOperation.executeOperation(PutIfAbsentOperation.java:23)
> at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:52)
> at org.infinispan.client.hotrod.impl.RemoteCacheImpl.putIfAbsent(RemoteCacheImpl.java:257)
> at org.infinispan.client.hotrod.impl.RemoteCacheSupport.putIfAbsent(RemoteCacheSupport.java:54)
> at HotRodTestClient.updateCache(HotRodTestClient.java:69)
> at HotRodTestClient.lambda$queuePut$0(HotRodTestClient.java:88)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-5803) Custom Key Results in ClassCastException in CacheLoader
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-5803?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño updated ISPN-5803:
-----------------------------------
Git Pull Request: https://github.com/infinispan/infinispan/pull/3753, https://github.com/infinispan/infinispan/pull/3773 (was: https://github.com/infinispan/infinispan/pull/3753)
> Custom Key Results in ClassCastException in CacheLoader
> -------------------------------------------------------
>
> Key: ISPN-5803
> URL: https://issues.jboss.org/browse/ISPN-5803
> Project: Infinispan
> Issue Type: Bug
> Components: JCache
> Affects Versions: 8.0.1.Final
> Reporter: Dan Siviter
> Assignee: Galder Zamarreño
> Fix For: 8.1.0.Beta1, 8.1.0.Final
>
>
> If a a JCache is created using a read-through {{javax.cache.integration.CacheLoader}} using a custom Serializable key (in this case {{acme.MyCache$MyKey}}) a {{ClassCastException}} is thrown when trying to pass the key value of {{org.infinispan.marshall.core.MarshalledValue}} instead of {{MarshalledValue#get()}} into the {{CacheLoader}} instance.
> {code}
> java.lang.ClassCastException: org.infinispan.marshall.core.MarshalledValue cannot be cast to acme.MyCache$MyKey
> at acme.MyCache$2.load(MyCache.java:1) [my-app-1.0.0-SNAPSHOT.jar:]
> at org.infinispan.jcache.embedded.JCacheLoaderAdapter.loadKey(JCacheLoaderAdapter.java:65) [infinispan-jcache-8.0.1.Final.jar:8.0.1.Final]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-4912) Switch REST store to better HTTP client library
by Antoine Toulme (JIRA)
[ https://issues.jboss.org/browse/ISPN-4912?page=com.atlassian.jira.plugin.... ]
Antoine Toulme commented on ISPN-4912:
--------------------------------------
I created a pull request with a change to use netty.
https://github.com/infinispan/infinispan/pull/3772
I have left quite a few things that are up for discussion, regarding the configuration options of Netty, identified as TODO comments in the code.
> Switch REST store to better HTTP client library
> -----------------------------------------------
>
> Key: ISPN-4912
> URL: https://issues.jboss.org/browse/ISPN-4912
> Project: Infinispan
> Issue Type: Enhancement
> Components: Loaders and Stores
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Fix For: 8.1.0.Final
>
>
> As a result of https://github.com/infinispan/infinispan/pull/3004, either OkHttp or Netty should do the trick.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-5869) Timeout in initial state replication when EAP cluster scaled up/down
by William DeCoste (JIRA)
[ https://issues.jboss.org/browse/ISPN-5869?page=com.atlassian.jira.plugin.... ]
William DeCoste commented on ISPN-5869:
---------------------------------------
Running SocketConnectionTimeoutTest ...
william-decostes-macbook-pro:tmp bdecoste$ ./run.sh
java.net.SocketTimeoutException: connect timed out
Connect took 0.314 s
Connect took 0.305 s
> Timeout in initial state replication when EAP cluster scaled up/down
> --------------------------------------------------------------------
>
> Key: ISPN-5869
> URL: https://issues.jboss.org/browse/ISPN-5869
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.2.11.Final
> Reporter: William DeCoste
> Assignee: Dan Berindei
> Attachments: down.sh, haproxy.cfg.down, haproxy.cfg.up, node1.log.gz, node2.log.gz, node3.log.gz, node4.log.gz, node5.log.gz, server.log, SocketConnectionTimeoutTest.java, up.sh
>
>
> When scaling an EAP cluster between 1 and 5 active instances with active traffic the below exception occurs fairly frequently which results in a dead EAP instance. This issue also results in a dead Pod when running in OpenShift. Active traffic is created state replication for HTTPSession and SFSB.
> 13:27:42,553 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 66) MSC000001: Failed to start service jboss.infinispan.web.default-host/sfsbTest: org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/sfsbTest: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [jboss-as-clustering-common-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.2.Final-redhat-1.jar:2.1.2.Final-redhat-1]
> Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:657)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:646)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:217)
> at org.infinispan.CacheImpl.start(CacheImpl.java:582)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:686)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:545)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:559)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:113)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:104)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) [jboss-as-clustering-common-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
> ... 4 more
> Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache default-host/sfsbTest on node2/web
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:216)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_45]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_45]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_45]
> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_45]
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
> ... 18 more
> 13:27:42,559 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014612: Operation ("deploy") failed - address: ([("deployment" => "sfsbTest.war")]) - failure description: {"JBAS014671: Failed services" => {"jboss.infinispan.web.default-host/sfsbTest" => "org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/sfsbTest: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache default-host/sfsbTest on node2/web"}}
> 13:27:42,606 INFO [org.jboss.as.server] (ServerService Thread Pool -- 35) JBAS015859: Deployed "sfsbTest.war" (runtime-name : "sfsbTest.war")
> 13:27:42,607 INFO [org.jboss.as.controller] (Controller Boot Thread) JBAS014774: Service status report
> JBAS014777: Services which failed to start: service jboss.infinispan.web.default-host/sfsbTest: org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/sfsbTest: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> 13:27:42,613 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:10000/management
> 13:27:42,613 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:10000
> 13:27:42,613 ERROR [org.jboss.as] (Controller Boot Thread) JBAS015875: JBoss EAP 6.4.0.GA (AS 7.5.0.Final-redhat-21) started (with errors) in 72734ms - Started 340 of 411 services (4 services failed or missing dependencies, 126 services are lazy, passive or on-demand)
> 13:27:42,850 INFO [org.jboss.as.jpa] (ServerService Thread Pool -- 23) JBAS011403: Stopping Persistence Unit Service 'sfsbTest.war#jpa-test'
> 13:27:42,858 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 23) JBAS010282: Stopped sfsbTest.war#jpa-test.org.jboss.jndiTest.TestEntity cache from hibernate container
> 13:27:42,872 INFO [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 23) ISPN000031: MBeans were successfully registered to the platform MBean server.
> 13:27:42,873 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 23) JBAS010281: Started local-query cache from hibernate container
> 13:27:42,873 INFO [org.hibernate.tool.hbm2ddl.SchemaExport] (ServerService Thread Pool -- 23) HHH000227: Running hbm2ddl schema export
> 13:27:42,878 INFO [org.hibernate.tool.hbm2ddl.SchemaExport] (ServerService Thread Pool -- 23) HHH000230: Schema export complete
> 13:27:42,894 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) JBAS010282: Stopped pending-puts cache from hibernate container
> 13:27:42,898 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) JBAS010282: Stopped local-query cache from hibernate container
> 13:27:42,899 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting and closing JGroups Channel
> 13:27:42,905 INFO [org.infinispan.eviction.PassivationManagerImpl] (ServerService Thread Pool -- 23) ISPN000029: Passivating all entries to disk
> 13:27:42,906 INFO [org.infinispan.eviction.PassivationManagerImpl] (ServerService Thread Pool -- 23) ISPN000030: Passivated 0 entries in 1 milliseconds
> 13:27:42,915 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 23) JBAS010282: Stopped repl cache from web container
> 13:27:42,918 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-4) ISPN000029: Passivating all entries to disk
> 13:27:42,920 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment sfsbTest.war (runtime-name: sfsbTest.war) in 85ms
> 13:27:43,263 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher
> 13:27:47,210 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-4) ISPN000030: Passivated 3955 entries in 4.29 seconds
> 13:27:47,223 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting and closing JGroups Channel
> 13:27:47,587 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher
> 13:27:47,662 INFO [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS015858: Undeployed "sfsbTest.war" (runtime-name: "sfsbTest.war")
> 13:27:47,664 INFO [org.jboss.as.controller] (DeploymentScanner-threads - 2) JBAS014774: Service status report
> JBAS014775: New missing/unsatisfied dependencies:
> service jboss.deployment.unit."sfsbTest.war".component.EntityTesterBean.START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component.StatefulBean1.START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component.StatelessBean1.START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component.StatelessBean2.START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component.TimerBean.START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."com.sun.faces.config.ConfigureListener".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."javax.faces.webapp.FacetTag".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."javax.servlet.jsp.jstl.tlv.ScriptFreeTLV".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."org.apache.catalina.servlets.DefaultServlet".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."org.apache.jasper.servlet.JspServlet".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."org.jboss.sfsbTest.SfsbServlet".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".jndiDependencyService (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest]
> service jboss.infinispan.web.default-host/sfsbTest (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest.session]
> service jboss.persistenceunit."sfsbTest.war#jpa-test" (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.web.deployment.default-host./sfsbTest (missing) dependents: [service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.web.deployment.default-host./sfsbTest.realm (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> JBAS014777: Services which failed to start: service jboss.infinispan.web.default-host/sfsbTest
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-5869) Timeout in initial state replication when EAP cluster scaled up/down
by William DeCoste (JIRA)
[ https://issues.jboss.org/browse/ISPN-5869?page=com.atlassian.jira.plugin.... ]
William DeCoste commented on ISPN-5869:
---------------------------------------
The original problem was in OpenShift with CLOUD-241 but I have been also testing by running locally, spinning up EAP normally - no containers. The attached up/down scripts will show how I am starting EAP.
I think the general problem is that I have an instance that is trying to replicate to 4 (now dead) instances + doing failure detection + getting 4 new instances added to the cluster + starting to replicate to the new instances
> Timeout in initial state replication when EAP cluster scaled up/down
> --------------------------------------------------------------------
>
> Key: ISPN-5869
> URL: https://issues.jboss.org/browse/ISPN-5869
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.2.11.Final
> Reporter: William DeCoste
> Assignee: Dan Berindei
> Attachments: down.sh, haproxy.cfg.down, haproxy.cfg.up, node1.log.gz, node2.log.gz, node3.log.gz, node4.log.gz, node5.log.gz, server.log, SocketConnectionTimeoutTest.java, up.sh
>
>
> When scaling an EAP cluster between 1 and 5 active instances with active traffic the below exception occurs fairly frequently which results in a dead EAP instance. This issue also results in a dead Pod when running in OpenShift. Active traffic is created state replication for HTTPSession and SFSB.
> 13:27:42,553 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 66) MSC000001: Failed to start service jboss.infinispan.web.default-host/sfsbTest: org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/sfsbTest: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [jboss-as-clustering-common-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.2.Final-redhat-1.jar:2.1.2.Final-redhat-1]
> Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:657)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:646)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:217)
> at org.infinispan.CacheImpl.start(CacheImpl.java:582)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:686)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:545)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:559)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:113)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:104)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) [jboss-as-clustering-common-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
> ... 4 more
> Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache default-host/sfsbTest on node2/web
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:216)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_45]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_45]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_45]
> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_45]
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
> ... 18 more
> 13:27:42,559 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014612: Operation ("deploy") failed - address: ([("deployment" => "sfsbTest.war")]) - failure description: {"JBAS014671: Failed services" => {"jboss.infinispan.web.default-host/sfsbTest" => "org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/sfsbTest: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache default-host/sfsbTest on node2/web"}}
> 13:27:42,606 INFO [org.jboss.as.server] (ServerService Thread Pool -- 35) JBAS015859: Deployed "sfsbTest.war" (runtime-name : "sfsbTest.war")
> 13:27:42,607 INFO [org.jboss.as.controller] (Controller Boot Thread) JBAS014774: Service status report
> JBAS014777: Services which failed to start: service jboss.infinispan.web.default-host/sfsbTest: org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/sfsbTest: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> 13:27:42,613 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:10000/management
> 13:27:42,613 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:10000
> 13:27:42,613 ERROR [org.jboss.as] (Controller Boot Thread) JBAS015875: JBoss EAP 6.4.0.GA (AS 7.5.0.Final-redhat-21) started (with errors) in 72734ms - Started 340 of 411 services (4 services failed or missing dependencies, 126 services are lazy, passive or on-demand)
> 13:27:42,850 INFO [org.jboss.as.jpa] (ServerService Thread Pool -- 23) JBAS011403: Stopping Persistence Unit Service 'sfsbTest.war#jpa-test'
> 13:27:42,858 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 23) JBAS010282: Stopped sfsbTest.war#jpa-test.org.jboss.jndiTest.TestEntity cache from hibernate container
> 13:27:42,872 INFO [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 23) ISPN000031: MBeans were successfully registered to the platform MBean server.
> 13:27:42,873 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 23) JBAS010281: Started local-query cache from hibernate container
> 13:27:42,873 INFO [org.hibernate.tool.hbm2ddl.SchemaExport] (ServerService Thread Pool -- 23) HHH000227: Running hbm2ddl schema export
> 13:27:42,878 INFO [org.hibernate.tool.hbm2ddl.SchemaExport] (ServerService Thread Pool -- 23) HHH000230: Schema export complete
> 13:27:42,894 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) JBAS010282: Stopped pending-puts cache from hibernate container
> 13:27:42,898 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) JBAS010282: Stopped local-query cache from hibernate container
> 13:27:42,899 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting and closing JGroups Channel
> 13:27:42,905 INFO [org.infinispan.eviction.PassivationManagerImpl] (ServerService Thread Pool -- 23) ISPN000029: Passivating all entries to disk
> 13:27:42,906 INFO [org.infinispan.eviction.PassivationManagerImpl] (ServerService Thread Pool -- 23) ISPN000030: Passivated 0 entries in 1 milliseconds
> 13:27:42,915 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 23) JBAS010282: Stopped repl cache from web container
> 13:27:42,918 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-4) ISPN000029: Passivating all entries to disk
> 13:27:42,920 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment sfsbTest.war (runtime-name: sfsbTest.war) in 85ms
> 13:27:43,263 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher
> 13:27:47,210 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-4) ISPN000030: Passivated 3955 entries in 4.29 seconds
> 13:27:47,223 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting and closing JGroups Channel
> 13:27:47,587 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher
> 13:27:47,662 INFO [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS015858: Undeployed "sfsbTest.war" (runtime-name: "sfsbTest.war")
> 13:27:47,664 INFO [org.jboss.as.controller] (DeploymentScanner-threads - 2) JBAS014774: Service status report
> JBAS014775: New missing/unsatisfied dependencies:
> service jboss.deployment.unit."sfsbTest.war".component.EntityTesterBean.START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component.StatefulBean1.START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component.StatelessBean1.START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component.StatelessBean2.START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component.TimerBean.START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."com.sun.faces.config.ConfigureListener".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."javax.faces.webapp.FacetTag".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."javax.servlet.jsp.jstl.tlv.ScriptFreeTLV".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."org.apache.catalina.servlets.DefaultServlet".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."org.apache.jasper.servlet.JspServlet".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."org.jboss.sfsbTest.SfsbServlet".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".jndiDependencyService (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest]
> service jboss.infinispan.web.default-host/sfsbTest (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest.session]
> service jboss.persistenceunit."sfsbTest.war#jpa-test" (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.web.deployment.default-host./sfsbTest (missing) dependents: [service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.web.deployment.default-host./sfsbTest.realm (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> JBAS014777: Services which failed to start: service jboss.infinispan.web.default-host/sfsbTest
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-5869) Timeout in initial state replication when EAP cluster scaled up/down
by William DeCoste (JIRA)
[ https://issues.jboss.org/browse/ISPN-5869?page=com.atlassian.jira.plugin.... ]
William DeCoste updated ISPN-5869:
----------------------------------
Attachment: node5.log.gz
node4.log.gz
node3.log.gz
node2.log.gz
node1.log.gz
Look for the following in node 3/4/5
[org.jboss.msc.service.fail] (ServerService Thread Pool -- 66) MSC000001: Failed to start service jboss.infinispan.web.default-host/sfsbTest: org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/sfsbTest: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> Timeout in initial state replication when EAP cluster scaled up/down
> --------------------------------------------------------------------
>
> Key: ISPN-5869
> URL: https://issues.jboss.org/browse/ISPN-5869
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.2.11.Final
> Reporter: William DeCoste
> Assignee: Dan Berindei
> Attachments: down.sh, haproxy.cfg.down, haproxy.cfg.up, node1.log.gz, node2.log.gz, node3.log.gz, node4.log.gz, node5.log.gz, server.log, SocketConnectionTimeoutTest.java, up.sh
>
>
> When scaling an EAP cluster between 1 and 5 active instances with active traffic the below exception occurs fairly frequently which results in a dead EAP instance. This issue also results in a dead Pod when running in OpenShift. Active traffic is created state replication for HTTPSession and SFSB.
> 13:27:42,553 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 66) MSC000001: Failed to start service jboss.infinispan.web.default-host/sfsbTest: org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/sfsbTest: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:91) [jboss-as-clustering-common-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]
> at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.2.Final-redhat-1.jar:2.1.2.Final-redhat-1]
> Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:205)
> at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:886)
> at org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:657)
> at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:646)
> at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549)
> at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:217)
> at org.infinispan.CacheImpl.start(CacheImpl.java:582)
> at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:686)
> at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:649)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:545)
> at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:559)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:113)
> at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:104)
> at org.jboss.as.clustering.infinispan.subsystem.CacheService.start(CacheService.java:78)
> at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86) [jboss-as-clustering-common-7.5.0.Final-redhat-21.jar:7.5.0.Final-redhat-21]
> ... 4 more
> Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache default-host/sfsbTest on node2/web
> at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:216)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_45]
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_45]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_45]
> at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_45]
> at org.infinispan.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:203)
> ... 18 more
> 13:27:42,559 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS014612: Operation ("deploy") failed - address: ([("deployment" => "sfsbTest.war")]) - failure description: {"JBAS014671: Failed services" => {"jboss.infinispan.web.default-host/sfsbTest" => "org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/sfsbTest: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> Caused by: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> Caused by: org.infinispan.CacheException: Initial state transfer timed out for cache default-host/sfsbTest on node2/web"}}
> 13:27:42,606 INFO [org.jboss.as.server] (ServerService Thread Pool -- 35) JBAS015859: Deployed "sfsbTest.war" (runtime-name : "sfsbTest.war")
> 13:27:42,607 INFO [org.jboss.as.controller] (Controller Boot Thread) JBAS014774: Service status report
> JBAS014777: Services which failed to start: service jboss.infinispan.web.default-host/sfsbTest: org.jboss.msc.service.StartException in service jboss.infinispan.web.default-host/sfsbTest: org.infinispan.CacheException: Unable to invoke method public void org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete() throws java.lang.InterruptedException on object of type StateTransferManagerImpl
> 13:27:42,613 INFO [org.jboss.as] (Controller Boot Thread) JBAS015961: Http management interface listening on http://127.0.0.1:10000/management
> 13:27:42,613 INFO [org.jboss.as] (Controller Boot Thread) JBAS015951: Admin console listening on http://127.0.0.1:10000
> 13:27:42,613 ERROR [org.jboss.as] (Controller Boot Thread) JBAS015875: JBoss EAP 6.4.0.GA (AS 7.5.0.Final-redhat-21) started (with errors) in 72734ms - Started 340 of 411 services (4 services failed or missing dependencies, 126 services are lazy, passive or on-demand)
> 13:27:42,850 INFO [org.jboss.as.jpa] (ServerService Thread Pool -- 23) JBAS011403: Stopping Persistence Unit Service 'sfsbTest.war#jpa-test'
> 13:27:42,858 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 23) JBAS010282: Stopped sfsbTest.war#jpa-test.org.jboss.jndiTest.TestEntity cache from hibernate container
> 13:27:42,872 INFO [org.infinispan.jmx.CacheJmxRegistration] (ServerService Thread Pool -- 23) ISPN000031: MBeans were successfully registered to the platform MBean server.
> 13:27:42,873 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 23) JBAS010281: Started local-query cache from hibernate container
> 13:27:42,873 INFO [org.hibernate.tool.hbm2ddl.SchemaExport] (ServerService Thread Pool -- 23) HHH000227: Running hbm2ddl schema export
> 13:27:42,878 INFO [org.hibernate.tool.hbm2ddl.SchemaExport] (ServerService Thread Pool -- 23) HHH000230: Schema export complete
> 13:27:42,894 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) JBAS010282: Stopped pending-puts cache from hibernate container
> 13:27:42,898 INFO [org.jboss.as.clustering.infinispan] (MSC service thread 1-5) JBAS010282: Stopped local-query cache from hibernate container
> 13:27:42,899 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000080: Disconnecting and closing JGroups Channel
> 13:27:42,905 INFO [org.infinispan.eviction.PassivationManagerImpl] (ServerService Thread Pool -- 23) ISPN000029: Passivating all entries to disk
> 13:27:42,906 INFO [org.infinispan.eviction.PassivationManagerImpl] (ServerService Thread Pool -- 23) ISPN000030: Passivated 0 entries in 1 milliseconds
> 13:27:42,915 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 23) JBAS010282: Stopped repl cache from web container
> 13:27:42,918 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-4) ISPN000029: Passivating all entries to disk
> 13:27:42,920 INFO [org.jboss.as.server.deployment] (MSC service thread 1-6) JBAS015877: Stopped deployment sfsbTest.war (runtime-name: sfsbTest.war) in 85ms
> 13:27:43,263 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-5) ISPN000082: Stopping the RpcDispatcher
> 13:27:47,210 INFO [org.infinispan.eviction.PassivationManagerImpl] (MSC service thread 1-4) ISPN000030: Passivated 3955 entries in 4.29 seconds
> 13:27:47,223 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting and closing JGroups Channel
> 13:27:47,587 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher
> 13:27:47,662 INFO [org.jboss.as.server] (DeploymentScanner-threads - 2) JBAS015858: Undeployed "sfsbTest.war" (runtime-name: "sfsbTest.war")
> 13:27:47,664 INFO [org.jboss.as.controller] (DeploymentScanner-threads - 2) JBAS014774: Service status report
> JBAS014775: New missing/unsatisfied dependencies:
> service jboss.deployment.unit."sfsbTest.war".component.EntityTesterBean.START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component.StatefulBean1.START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component.StatelessBean1.START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component.StatelessBean2.START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component.TimerBean.START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."com.sun.faces.config.ConfigureListener".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."javax.faces.webapp.FacetTag".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."javax.servlet.jsp.jstl.tlv.PermittedTaglibsTLV".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."javax.servlet.jsp.jstl.tlv.ScriptFreeTLV".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."org.apache.catalina.servlets.DefaultServlet".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."org.apache.jasper.servlet.JspServlet".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".component."org.jboss.sfsbTest.SfsbServlet".START (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.deployment.unit."sfsbTest.war".jndiDependencyService (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest]
> service jboss.infinispan.web.default-host/sfsbTest (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest.session]
> service jboss.persistenceunit."sfsbTest.war#jpa-test" (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.web.deployment.default-host./sfsbTest (missing) dependents: [service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> service jboss.web.deployment.default-host./sfsbTest.realm (missing) dependents: [service jboss.web.deployment.default-host./sfsbTest, service jboss.deployment.unit."sfsbTest.war".deploymentCompleteService]
> JBAS014777: Services which failed to start: service jboss.infinispan.web.default-host/sfsbTest
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-5876) Pre-commit cache invalidation creates stale cache vulnerability
by Stephen Fikes (JIRA)
[ https://issues.jboss.org/browse/ISPN-5876?page=com.atlassian.jira.plugin.... ]
Stephen Fikes updated ISPN-5876:
--------------------------------
Description:
In a cluster where Infinispan serves as the level 2 cache for Hibernate (configured for invalidation), because invalidation requests for modified entities are sent *before* database commit, it is possible for nodes receiving the invalidation request to perform eviction and then (due to "local" read requests) reload the evicted entities prior to the time the database commit takes place in the server where the entity was modified.
Consequently, other servers in the cluster may contain data that remains stale until a subsequent change in another server or until the entity times out from lack of use.
It isn't easy to write a testcase for this - it required manual intervention to reproduce - but can be seen with any entity class, cluster, etc. (at least using Oracle - results may vary with specific databases) so I've not attached a testcase. The issue can be seen/understood by code inspection (i.e. the timing of invalidation vs. database commit). That said, my test consisted of a two node cluster and I used Byteman rules to delay database commit of a change to an entity (with an optimistic version property) long enough in "server 1" for eviction to complete and a subsequent re-read (by a worker thread on behalf of an EJB) to take place in "server 2". Following the re-read in "server 2", I the database commit proceeds in "server 1" and "server 2" now has a stale copy of the entity in cache.
One option is pessimistic locking which will block any read attempt until the DB commit completes. It is not feasible, however, for many applications to use pessimistic locking for all reads as this can have a severe impact on concurrency - and is the reason for using optimistic version control. But due to the early timing of invalidation broadcast (*before* database commit, while the data is not yet stale), optimistic locking is insufficient to guard against "permanently" stale data. We did see that some databases default to blocking repeatable reads even outside of transactions and without explicit lock requests. Oracle does not provide such a mode. So, all reads must be implemented to use pessimistic locks (which must be enclosed in explicit transactions - (b)locking reads are disallowed when autocommit=true in Oracle) and this could require significant effort (re-writes) to use pessimistic reads throughout - in addition to the performance issues this can introduce.
If broadcast of an invalidation message always occurs *after* database commit, optimistic control attributes are sufficient to block attempts to write stale data and though a few failures may occur (as they would in a single server with multiple active threads), it can be known that the stale data will be removed in some finite period.
was:
In a cluster where Infinispan serves as the level 2 cache for Hibernate (configured for invalidation), because invalidation requests for modified entities are sent *before* database commit, it is possible for nodes receiving the invalidation request to perform eviction and then (due to "local" read requests) reload the evicted entities prior to the time the database commit takes place in the server where the entity was modified.
Consequently, other servers in the cluster may contain data that remains stale until a subsequent change in another server or until the entity times out from lack of use.
It isn't easy to write a testcase for this - it requires lots of manual intervention - but it can be seen with any entity class, cluster, etc. (at least using Oracle - results may be different in some databases) so I've not attached a testcase. The issue is quite general and can be seen/understood by code inspection (i.e. the timing of invalidation vs. database commit). That said, I set up a two node cluster and I used Byteman rules to delay database commit of a change to an entity (with an optimistic version property) long enough in "server 1" for eviction and re-load to take place in "server 2". Following the reload in "server 2", I the database commit proceeds in "server 1" and "server 2" now has a stale copy of the entity in cache. This can be seen with any entity.
One option is pessimistic locking which will block any read attempt until the DB commit completes. It is not feasible, however, for many applications to use pessimistic locking for all reads as this can have a severe impact on concurrency - and is the reason that optimistic locking was created. But due to the early timing of invalidation broadcast (*before* database commit), optimistic locking is insufficient to guard against "permanently" stale data. We did see that some databases default to blocking repeatable reads even outside of transactions and without explicit lock requests. Oracle does not provide such a mode. So, all reads must be implemented to use pessimistic locks (which must be enclosed in explicit transactions - (b)locking reads are disallowed when autocommit=true in Oracle) and this could require significant effort (re-writes) to use pessimistic reads throughout.
If broadcast of an invalidation message always occurs *after* database commit, optimistic control attributes block attempts to write stale data and though a few failures may occur, it can be known that the stale data will be removed in some finite period.
> Pre-commit cache invalidation creates stale cache vulnerability
> ---------------------------------------------------------------
>
> Key: ISPN-5876
> URL: https://issues.jboss.org/browse/ISPN-5876
> Project: Infinispan
> Issue Type: Bug
> Components: Eviction
> Affects Versions: 5.2.7.Final
> Reporter: Stephen Fikes
> Assignee: Galder Zamarreño
>
> In a cluster where Infinispan serves as the level 2 cache for Hibernate (configured for invalidation), because invalidation requests for modified entities are sent *before* database commit, it is possible for nodes receiving the invalidation request to perform eviction and then (due to "local" read requests) reload the evicted entities prior to the time the database commit takes place in the server where the entity was modified.
> Consequently, other servers in the cluster may contain data that remains stale until a subsequent change in another server or until the entity times out from lack of use.
> It isn't easy to write a testcase for this - it required manual intervention to reproduce - but can be seen with any entity class, cluster, etc. (at least using Oracle - results may vary with specific databases) so I've not attached a testcase. The issue can be seen/understood by code inspection (i.e. the timing of invalidation vs. database commit). That said, my test consisted of a two node cluster and I used Byteman rules to delay database commit of a change to an entity (with an optimistic version property) long enough in "server 1" for eviction to complete and a subsequent re-read (by a worker thread on behalf of an EJB) to take place in "server 2". Following the re-read in "server 2", I the database commit proceeds in "server 1" and "server 2" now has a stale copy of the entity in cache.
> One option is pessimistic locking which will block any read attempt until the DB commit completes. It is not feasible, however, for many applications to use pessimistic locking for all reads as this can have a severe impact on concurrency - and is the reason for using optimistic version control. But due to the early timing of invalidation broadcast (*before* database commit, while the data is not yet stale), optimistic locking is insufficient to guard against "permanently" stale data. We did see that some databases default to blocking repeatable reads even outside of transactions and without explicit lock requests. Oracle does not provide such a mode. So, all reads must be implemented to use pessimistic locks (which must be enclosed in explicit transactions - (b)locking reads are disallowed when autocommit=true in Oracle) and this could require significant effort (re-writes) to use pessimistic reads throughout - in addition to the performance issues this can introduce.
> If broadcast of an invalidation message always occurs *after* database commit, optimistic control attributes are sufficient to block attempts to write stale data and though a few failures may occur (as they would in a single server with multiple active threads), it can be known that the stale data will be removed in some finite period.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-5876) Pre-commit cache invalidation creates stale cache vulnerability
by Stephen Fikes (JIRA)
[ https://issues.jboss.org/browse/ISPN-5876?page=com.atlassian.jira.plugin.... ]
Stephen Fikes updated ISPN-5876:
--------------------------------
Description:
In a cluster where Infinispan serves as the level 2 cache for Hibernate (configured for invalidation), because invalidation requests for modified entities are sent *before* database commit, it is possible for nodes receiving the invalidation request to perform eviction and then (due to "local" read requests) reload the evicted entities prior to the time the database commit takes place in the server where the entity was modified.
Consequently, other servers in the cluster may contain data that remains stale until a subsequent change in another server or until the entity times out from lack of use.
It isn't easy to write a testcase for this - it requires lots of manual intervention - but it can be seen with any entity class, cluster, etc. (at least using Oracle - results may be different in some databases) so I've not attached a testcase. The issue is quite general and can be seen/understood by code inspection (i.e. the timing of invalidation vs. database commit). That said, I set up a two node cluster and I used Byteman rules to delay database commit of a change to an entity (with an optimistic version property) long enough in "server 1" for eviction and re-load to take place in "server 2". Following the reload in "server 2", I the database commit proceeds in "server 1" and "server 2" now has a stale copy of the entity in cache. This can be seen with any entity.
One option is pessimistic locking which will block any read attempt until the DB commit completes. It is not feasible, however, for many applications to use pessimistic locking for all reads as this can have a severe impact on concurrency - and is the reason that optimistic locking was created. But due to the early timing of invalidation broadcast (*before* database commit), optimistic locking is insufficient to guard against "permanently" stale data. We did see that some databases default to blocking repeatable reads even outside of transactions and without explicit lock requests. Oracle does not provide such a mode. So, all reads must be implemented to use pessimistic locks (which must be enclosed in explicit transactions - (b)locking reads are disallowed when autocommit=true in Oracle) and this could require significant effort (re-writes) to use pessimistic reads throughout.
If broadcast of an invalidation message always occurs *after* database commit, optimistic control attributes block attempts to write stale data and though a few failures may occur, it can be known that the stale data will be removed in some finite period.
was:
In a cluster where Infinispan serves as the level 2 cache for Hibernate (configured for invalidation), because invalidation requests for modified entities are sent *before* database commit, it is possible for nodes receiving the invalidation request to perform eviction and then (due to "local" read requests) reload the evicted entities prior to the time the database commit takes place in the server where the entity was modified.
Consequently, other servers in the cluster may contain data that remains stale until a subsequent change in another server or until the entity times out from lack of use.
It isn't easy to write a testcase for this - it requires lots of manual intervention - but it can be seen with any entity class, cluster, etc. (at least using Oracle - results may be different in some databases) so I've not attached a testcase. The issue is quite general and can be seen/understood by code inspection (i.e. the timing of invalidation vs. database commit). That said, I set up a two node cluster and I used Byteman rules to delay database commit of a change to an entity (with an optimistic version property) long enough in "server 1" for eviction and re-load to take place in "server 2". Following the reload in "server 2", I the database commit proceeds in "server 1" and "server 2" now has a stale copy of the entity in cache. This can be seen with any entity.
One option is pessimistic locking which will block any read attempt until the DB commit completes. It is not feasible, however, for many applications to use pessimistic locking for all reads as this can have a severe impact on concurrency - and is the reason that optimistic locking was created. But due to the early timing of invalidation broadcast (*before* database commit), optimistic locking is insufficient to guard against "permanently" stale data. We did see that some databases default to blocking repeatable reads even outside of transactions and without explicit lock requests. Oracle does not provide such a mode. So, all reads must be implemented to use pessimistic locks (which must be enclosed in explicit transactions - (b)locking reads are disallowed when autocommit=true) and this could require significant effort (re-writes) to use pessimistic reads throughout.
If broadcast of an invalidation message always occurs *after* database commit, optimistic control attributes block attempts to write stale data and though a few failures may occur, it can be known that the stale data will be removed in some finite period.
> Pre-commit cache invalidation creates stale cache vulnerability
> ---------------------------------------------------------------
>
> Key: ISPN-5876
> URL: https://issues.jboss.org/browse/ISPN-5876
> Project: Infinispan
> Issue Type: Bug
> Components: Eviction
> Affects Versions: 5.2.7.Final
> Reporter: Stephen Fikes
> Assignee: Galder Zamarreño
>
> In a cluster where Infinispan serves as the level 2 cache for Hibernate (configured for invalidation), because invalidation requests for modified entities are sent *before* database commit, it is possible for nodes receiving the invalidation request to perform eviction and then (due to "local" read requests) reload the evicted entities prior to the time the database commit takes place in the server where the entity was modified.
> Consequently, other servers in the cluster may contain data that remains stale until a subsequent change in another server or until the entity times out from lack of use.
> It isn't easy to write a testcase for this - it requires lots of manual intervention - but it can be seen with any entity class, cluster, etc. (at least using Oracle - results may be different in some databases) so I've not attached a testcase. The issue is quite general and can be seen/understood by code inspection (i.e. the timing of invalidation vs. database commit). That said, I set up a two node cluster and I used Byteman rules to delay database commit of a change to an entity (with an optimistic version property) long enough in "server 1" for eviction and re-load to take place in "server 2". Following the reload in "server 2", I the database commit proceeds in "server 1" and "server 2" now has a stale copy of the entity in cache. This can be seen with any entity.
> One option is pessimistic locking which will block any read attempt until the DB commit completes. It is not feasible, however, for many applications to use pessimistic locking for all reads as this can have a severe impact on concurrency - and is the reason that optimistic locking was created. But due to the early timing of invalidation broadcast (*before* database commit), optimistic locking is insufficient to guard against "permanently" stale data. We did see that some databases default to blocking repeatable reads even outside of transactions and without explicit lock requests. Oracle does not provide such a mode. So, all reads must be implemented to use pessimistic locks (which must be enclosed in explicit transactions - (b)locking reads are disallowed when autocommit=true in Oracle) and this could require significant effort (re-writes) to use pessimistic reads throughout.
> If broadcast of an invalidation message always occurs *after* database commit, optimistic control attributes block attempts to write stale data and though a few failures may occur, it can be known that the stale data will be removed in some finite period.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-5876) Pre-commit cache invalidation creates stale cache vulnerability
by Stephen Fikes (JIRA)
[ https://issues.jboss.org/browse/ISPN-5876?page=com.atlassian.jira.plugin.... ]
Stephen Fikes updated ISPN-5876:
--------------------------------
Description:
In a cluster where Infinispan serves as the level 2 cache for Hibernate (configured for invalidation), because invalidation requests for modified entities are sent *before* database commit, it is possible for nodes receiving the invalidation request to perform eviction and then (due to "local" read requests) reload the evicted entities prior to the time the database commit takes place in the server where the entity was modified.
Consequently, other servers in the cluster may contain data that remains stale until a subsequent change in another server or until the entity times out from lack of use.
It isn't easy to write a testcase for this - it requires lots of manual intervention - but it can be seen with any entity class, cluster, etc. (at least using Oracle - results may be different in some databases) so I've not attached a testcase. The issue is quite general and can be seen/understood by code inspection (i.e. the timing of invalidation vs. database commit). That said, I set up a two node cluster and I used Byteman rules to delay database commit of a change to an entity (with an optimistic version property) long enough in "server 1" for eviction and re-load to take place in "server 2". Following the reload in "server 2", I the database commit proceeds in "server 1" and "server 2" now has a stale copy of the entity in cache. This can be seen with any entity.
One option is pessimistic locking which will block any read attempt until the DB commit completes. It is not feasible, however, for many applications to use pessimistic locking for all reads as this can have a severe impact on concurrency - and is the reason that optimistic locking was created. But due to the early timing of invalidation broadcast (*before* database commit), optimistic locking is insufficient to guard against "permanently" stale data. We did see that some databases default to blocking repeatable reads even outside of transactions and without explicit lock requests. Oracle does not provide such a mode. So, all reads must be implemented to use pessimistic locks (which must be enclosed in explicit transactions - (b)locking reads are disallowed when autocommit=true) and this could require significant effort (re-writes) to use pessimistic reads throughout.
If broadcast of an invalidation message always occurs *after* database commit, optimistic control attributes block attempts to write stale data and though a few failures may occur, it can be known that the stale data will be removed in some finite period.
was:
In a cluster where Infinispan serves as the level 2 cache for Hibernate (configured for invalidation), because invalidation requests for modified entities are sent *before* database commit, it is possible for nodes receiving the invalidation request to perform eviction and then (due to "local" read requests) reload the evicted entities prior to the time the database commit takes place in the server where the entity was modified.
Consequently, other servers in the cluster may contain data that remains stale until a subsequent change in another server or until the entity times out from lack of use.
It isn't easy to write a testcase for this - it requires lots of manual intervention - but it can be seen with any entity class, cluster, etc. (at least using Oracle - results may be different in some databases) so I'm not attaching anything. The issue is quite general and can be seen/understood by code inspection (i.e. the timing of invalidation vs. database commit). That said, I set up a two node cluster and I used Byteman rules to delay database commit of a change to an entity (with an optimistic version property) long enough in "server 1" for eviction and re-load to take place in "server 2". Following the reload in "server 2", I the database commit proceeds in "server 1" and "server 2" now has a stale copy of the entity in cache. This can be seen with any entity.
One option is pessimistic locking which will block any read attempt until the DB commit completes. It is not feasible, however, for many applications to use pessimistic locking for all reads as this can have a severe impact on concurrency - and is the reason that optimistic locking was created. But due to the early timing of invalidation broadcast (*before* database commit), optimistic locking is insufficient to guard against "permanently" stale data. We did see that some databases default to blocking repeatable reads even outside of transactions and without explicit lock requests. Oracle does not provide such a mode. So, all reads must be implemented to use pessimistic locks (which must be enclosed in explicit transactions - (b)locking reads are disallowed when autocommit=true) and this could require significant effort (re-writes) to use pessimistic reads throughout.
If broadcast of an invalidation message always occurs *after* database commit, optimistic control attributes block attempts to write stale data and though a few failures may occur, it can be known that the stale data will be removed in some finite period.
> Pre-commit cache invalidation creates stale cache vulnerability
> ---------------------------------------------------------------
>
> Key: ISPN-5876
> URL: https://issues.jboss.org/browse/ISPN-5876
> Project: Infinispan
> Issue Type: Bug
> Components: Eviction
> Affects Versions: 5.2.7.Final
> Reporter: Stephen Fikes
> Assignee: Galder Zamarreño
>
> In a cluster where Infinispan serves as the level 2 cache for Hibernate (configured for invalidation), because invalidation requests for modified entities are sent *before* database commit, it is possible for nodes receiving the invalidation request to perform eviction and then (due to "local" read requests) reload the evicted entities prior to the time the database commit takes place in the server where the entity was modified.
> Consequently, other servers in the cluster may contain data that remains stale until a subsequent change in another server or until the entity times out from lack of use.
> It isn't easy to write a testcase for this - it requires lots of manual intervention - but it can be seen with any entity class, cluster, etc. (at least using Oracle - results may be different in some databases) so I've not attached a testcase. The issue is quite general and can be seen/understood by code inspection (i.e. the timing of invalidation vs. database commit). That said, I set up a two node cluster and I used Byteman rules to delay database commit of a change to an entity (with an optimistic version property) long enough in "server 1" for eviction and re-load to take place in "server 2". Following the reload in "server 2", I the database commit proceeds in "server 1" and "server 2" now has a stale copy of the entity in cache. This can be seen with any entity.
> One option is pessimistic locking which will block any read attempt until the DB commit completes. It is not feasible, however, for many applications to use pessimistic locking for all reads as this can have a severe impact on concurrency - and is the reason that optimistic locking was created. But due to the early timing of invalidation broadcast (*before* database commit), optimistic locking is insufficient to guard against "permanently" stale data. We did see that some databases default to blocking repeatable reads even outside of transactions and without explicit lock requests. Oracle does not provide such a mode. So, all reads must be implemented to use pessimistic locks (which must be enclosed in explicit transactions - (b)locking reads are disallowed when autocommit=true) and this could require significant effort (re-writes) to use pessimistic reads throughout.
> If broadcast of an invalidation message always occurs *after* database commit, optimistic control attributes block attempts to write stale data and though a few failures may occur, it can be known that the stale data will be removed in some finite period.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
- 1
- 0
[JBoss JIRA] (ISPN-5876) Pre-commit cache invalidation creates stale cache vulnerability
by Stephen Fikes (JIRA)
[ https://issues.jboss.org/browse/ISPN-5876?page=com.atlassian.jira.plugin.... ]
Stephen Fikes updated ISPN-5876:
--------------------------------
Description:
In a cluster where Infinispan serves as the level 2 cache for Hibernate (configured for invalidation), because invalidation requests for modified entities are sent *before* database commit, it is possible for nodes receiving the invalidation request to perform eviction and then (due to "local" read requests) reload the evicted entities prior to the time the database commit takes place in the server where the entity was modified.
Consequently, other servers in the cluster may contain data that remains stale until a subsequent change in another server or until the entity times out from lack of use.
It isn't easy to write a testcase for this - it requires lots of manual intervention - but it can be seen with any entity class, cluster, etc. (at least using Oracle - results may be different in some databases) so I'm not attaching anything. The issue is quite general and can be seen/understood by code inspection (i.e. the timing of invalidation vs. database commit). That said, I set up a two node cluster and I used Byteman rules to delay database commit of a change to an entity (with an optimistic version property) long enough in "server 1" for eviction and re-load to take place in "server 2". Following the reload in "server 2", I the database commit proceeds in "server 1" and "server 2" now has a stale copy of the entity in cache. This can be seen with any entity.
One option is pessimistic locking which will block any read attempt until the DB commit completes. It is not feasible, however, for many applications to use pessimistic locking for all reads as this can have a severe impact on concurrency - and is the reason that optimistic locking was created. But due to the early timing of invalidation broadcast (*before* database commit), optimistic locking is insufficient to guard against "permanently" stale data. We did see that some databases default to blocking repeatable reads even outside of transactions and without explicit lock requests. Oracle does not provide such a mode. So, all reads must be implemented to use pessimistic locks (which must be enclosed in explicit transactions - (b)locking reads are disallowed when autocommit=true) and this could require significant effort (re-writes) to use pessimistic reads throughout.
If broadcast of an invalidation message always occurs *after* database commit, optimistic control attributes block attempts to write stale data and though a few failures may occur, it can be known that the stale data will be removed in some finite period.
was:
In a cluster where Infinispan serves as the level 2 cache for Hibernate (configured for invalidation), because invalidation requests for modified entities are sent *before* database commit, it is possible for nodes receiving the invalidation request to perform eviction and then (due to "local" read requests) reload the evicted entities prior to the time the database commit takes place in the server where the entity was modified.
Consequently, other servers in the cluster may contain data that remains stale until a subsequent change in another server or until the entity times out from lack of use.
It isn't easy to write a testcase for this - it requires lots of manual intervention - but it can be seen with any entity class, cluster, database, etc. so I'm not attaching anything. The issue is quite general and can be seen/understood by code inspection (i.e. the timing of invalidation vs. database commit). That said, I set up a two node cluster and I used Byteman rules to delay database commit of a change to an entity (with an optimistic version property) long enough in "server 1" for eviction and re-load to take place in "server 2". Following the reload in "server 2", I the database commit proceeds in "server 1" and "server 2" now has a stale copy of the entity in cache. This can be seen with any entity.
One option is pessimistic locking which will block any read attempt until the DB commit completes. It is not feasible, however, for many applications to use pessimistic locking for all reads as this can have a severe impact on concurrency - and is the reason that optimistic locking was created. But due to the early timing of invalidation broadcast (*before* database commit), optimistic locking is insufficient to guard against "permanently" stale data. We did see that some databases default to blocking repeatable reads even outside of transactions and without explicit lock requests. Oracle does not provide such a mode. So, all reads must be implemented to use pessimistic locks (which must be enclosed in explicit transactions - (b)locking reads are disallowed when autocommit=true) and this could require significant effort (re-writes) to use pessimistic reads throughout.
If broadcast of an invalidation message always occurs *after* database commit, optimistic control attributes block attempts to write stale data and though a few failures may occur, it can be known that the stale data will be removed in some finite period.
> Pre-commit cache invalidation creates stale cache vulnerability
> ---------------------------------------------------------------
>
> Key: ISPN-5876
> URL: https://issues.jboss.org/browse/ISPN-5876
> Project: Infinispan
> Issue Type: Bug
> Components: Eviction
> Affects Versions: 5.2.7.Final
> Reporter: Stephen Fikes
> Assignee: Galder Zamarreño
>
> In a cluster where Infinispan serves as the level 2 cache for Hibernate (configured for invalidation), because invalidation requests for modified entities are sent *before* database commit, it is possible for nodes receiving the invalidation request to perform eviction and then (due to "local" read requests) reload the evicted entities prior to the time the database commit takes place in the server where the entity was modified.
> Consequently, other servers in the cluster may contain data that remains stale until a subsequent change in another server or until the entity times out from lack of use.
> It isn't easy to write a testcase for this - it requires lots of manual intervention - but it can be seen with any entity class, cluster, etc. (at least using Oracle - results may be different in some databases) so I'm not attaching anything. The issue is quite general and can be seen/understood by code inspection (i.e. the timing of invalidation vs. database commit). That said, I set up a two node cluster and I used Byteman rules to delay database commit of a change to an entity (with an optimistic version property) long enough in "server 1" for eviction and re-load to take place in "server 2". Following the reload in "server 2", I the database commit proceeds in "server 1" and "server 2" now has a stale copy of the entity in cache. This can be seen with any entity.
> One option is pessimistic locking which will block any read attempt until the DB commit completes. It is not feasible, however, for many applications to use pessimistic locking for all reads as this can have a severe impact on concurrency - and is the reason that optimistic locking was created. But due to the early timing of invalidation broadcast (*before* database commit), optimistic locking is insufficient to guard against "permanently" stale data. We did see that some databases default to blocking repeatable reads even outside of transactions and without explicit lock requests. Oracle does not provide such a mode. So, all reads must be implemented to use pessimistic locks (which must be enclosed in explicit transactions - (b)locking reads are disallowed when autocommit=true) and this could require significant effort (re-writes) to use pessimistic reads throughout.
> If broadcast of an invalidation message always occurs *after* database commit, optimistic control attributes block attempts to write stale data and though a few failures may occur, it can be known that the stale data will be removed in some finite period.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
9 years, 3 months
- 1
- 0