[JBoss JIRA] (ISPN-10924) JPA cache store (with async enabled) will log Exceptions in DEBUG for deleteBatch and is not able to remove entries from the store
by Ryan Emerson (Jira)
[ https://issues.jboss.org/browse/ISPN-10924?page=com.atlassian.jira.plugin... ]
Work on ISPN-10924 started by Ryan Emerson.
-------------------------------------------
> JPA cache store (with async enabled) will log Exceptions in DEBUG for deleteBatch and is not able to remove entries from the store
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-10924
> URL: https://issues.jboss.org/browse/ISPN-10924
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 9.4.16.Final, 10.0.1.Final, 10.1.0.Beta1
> Environment: Embedded mode with configured ASYNC JPA CacheStore
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Priority: Major
>
> If the cache is configured with a persistence of JPAStore in async mode each modification will log a DEBUG message like followed
> Oct 22, 2019 1:51:08 PM org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreProcessor retryWork
> DEBUG: Failed to process async modifications
> org.infinispan.persistence.jpa.JpaStoreException: Exception caught in deleteBatch()
> at org.infinispan.persistence.jpa.JpaStore.deleteBatch(JpaStore.java:357)
> at org.infinispan.persistence.async.AsyncCacheWriter.applyModificationsSync(AsyncCacheWriter.java:234)
> at org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreProcessor.retryWork(AsyncCacheWriter.java:463)
> at org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreProcessor.run(AsyncCacheWriter.java:423)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: Parameter value [org.infinispan.persistence.async.AsyncCacheWriter$$Lambda$136/1340865907@4c7c351c] did not match expected type [java.lang.String (n/a)]
> at org.hibernate.query.spi.QueryParameterBindingValidator.validate(QueryParameterBindingValidator.java:54)
> at org.hibernate.query.spi.QueryParameterBindingValidator.validate(QueryParameterBindingValidator.java:27)
> at org.hibernate.query.internal.QueryParameterBindingImpl.validate(QueryParameterBindingImpl.java:90)
> at org.hibernate.query.internal.QueryParameterBindingImpl.setBindValue(QueryParameterBindingImpl.java:55)
> at org.hibernate.query.internal.AbstractProducedQuery.setParameter(AbstractProducedQuery.java:493)
> at org.hibernate.query.internal.AbstractProducedQuery.setParameter(AbstractProducedQuery.java:106)
> at org.hibernate.query.criteria.internal.compile.CriteriaCompiler$1$1.bind(CriteriaCompiler.java:119)
> at org.hibernate.query.criteria.internal.AbstractManipulationCriteriaQuery$1.buildCompiledQuery(AbstractManipulationCriteriaQuery.java:135)
> at org.hibernate.query.criteria.internal.compile.CriteriaCompiler.compile(CriteriaCompiler.java:149)
> at org.hibernate.internal.SessionImpl.createQuery(SessionImpl.java:3724)
> at org.hibernate.internal.SessionImpl.createQuery(SessionImpl.java:207)
> at org.infinispan.persistence.jpa.JpaStore.deleteBatch(JpaStore.java:341)
> ... 6 more
> If entries should be removed this is only done in memory but the store will fail ASYNC.
> Because of the async contract there is only a warning wihtout any exact hint:
> WARN: ISPN000053: Unable to process some async modifications after 10 retries!
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 11 months
[JBoss JIRA] (ISPN-10924) JPA cache store (with async enabled) will log Exceptions in DEBUG for deleteBatch and is not able to remove entries from the store
by Ryan Emerson (Jira)
[ https://issues.jboss.org/browse/ISPN-10924?page=com.atlassian.jira.plugin... ]
Ryan Emerson updated ISPN-10924:
--------------------------------
Status: Open (was: New)
> JPA cache store (with async enabled) will log Exceptions in DEBUG for deleteBatch and is not able to remove entries from the store
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-10924
> URL: https://issues.jboss.org/browse/ISPN-10924
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 9.4.16.Final, 10.0.1.Final, 10.1.0.Beta1
> Environment: Embedded mode with configured ASYNC JPA CacheStore
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Priority: Major
>
> If the cache is configured with a persistence of JPAStore in async mode each modification will log a DEBUG message like followed
> Oct 22, 2019 1:51:08 PM org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreProcessor retryWork
> DEBUG: Failed to process async modifications
> org.infinispan.persistence.jpa.JpaStoreException: Exception caught in deleteBatch()
> at org.infinispan.persistence.jpa.JpaStore.deleteBatch(JpaStore.java:357)
> at org.infinispan.persistence.async.AsyncCacheWriter.applyModificationsSync(AsyncCacheWriter.java:234)
> at org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreProcessor.retryWork(AsyncCacheWriter.java:463)
> at org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreProcessor.run(AsyncCacheWriter.java:423)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: Parameter value [org.infinispan.persistence.async.AsyncCacheWriter$$Lambda$136/1340865907@4c7c351c] did not match expected type [java.lang.String (n/a)]
> at org.hibernate.query.spi.QueryParameterBindingValidator.validate(QueryParameterBindingValidator.java:54)
> at org.hibernate.query.spi.QueryParameterBindingValidator.validate(QueryParameterBindingValidator.java:27)
> at org.hibernate.query.internal.QueryParameterBindingImpl.validate(QueryParameterBindingImpl.java:90)
> at org.hibernate.query.internal.QueryParameterBindingImpl.setBindValue(QueryParameterBindingImpl.java:55)
> at org.hibernate.query.internal.AbstractProducedQuery.setParameter(AbstractProducedQuery.java:493)
> at org.hibernate.query.internal.AbstractProducedQuery.setParameter(AbstractProducedQuery.java:106)
> at org.hibernate.query.criteria.internal.compile.CriteriaCompiler$1$1.bind(CriteriaCompiler.java:119)
> at org.hibernate.query.criteria.internal.AbstractManipulationCriteriaQuery$1.buildCompiledQuery(AbstractManipulationCriteriaQuery.java:135)
> at org.hibernate.query.criteria.internal.compile.CriteriaCompiler.compile(CriteriaCompiler.java:149)
> at org.hibernate.internal.SessionImpl.createQuery(SessionImpl.java:3724)
> at org.hibernate.internal.SessionImpl.createQuery(SessionImpl.java:207)
> at org.infinispan.persistence.jpa.JpaStore.deleteBatch(JpaStore.java:341)
> ... 6 more
> If entries should be removed this is only done in memory but the store will fail ASYNC.
> Because of the async contract there is only a warning wihtout any exact hint:
> WARN: ISPN000053: Unable to process some async modifications after 10 retries!
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 11 months
[JBoss JIRA] (ISPN-10924) JPA cache store (with async enabled) will log Exceptions in DEBUG for deleteBatch and is not able to remove entries from the store
by Ryan Emerson (Jira)
[ https://issues.jboss.org/browse/ISPN-10924?page=com.atlassian.jira.plugin... ]
Ryan Emerson updated ISPN-10924:
--------------------------------
Security: (was: Red Hat Internal)
> JPA cache store (with async enabled) will log Exceptions in DEBUG for deleteBatch and is not able to remove entries from the store
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: ISPN-10924
> URL: https://issues.jboss.org/browse/ISPN-10924
> Project: Infinispan
> Issue Type: Bug
> Components: Loaders and Stores
> Affects Versions: 9.4.16.Final, 10.0.1.Final, 10.1.0.Beta1
> Environment: Embedded mode with configured ASYNC JPA CacheStore
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Priority: Major
>
> If the cache is configured with a persistence of JPAStore in async mode each modification will log a DEBUG message like followed
> Oct 22, 2019 1:51:08 PM org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreProcessor retryWork
> DEBUG: Failed to process async modifications
> org.infinispan.persistence.jpa.JpaStoreException: Exception caught in deleteBatch()
> at org.infinispan.persistence.jpa.JpaStore.deleteBatch(JpaStore.java:357)
> at org.infinispan.persistence.async.AsyncCacheWriter.applyModificationsSync(AsyncCacheWriter.java:234)
> at org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreProcessor.retryWork(AsyncCacheWriter.java:463)
> at org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreProcessor.run(AsyncCacheWriter.java:423)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.IllegalArgumentException: Parameter value [org.infinispan.persistence.async.AsyncCacheWriter$$Lambda$136/1340865907@4c7c351c] did not match expected type [java.lang.String (n/a)]
> at org.hibernate.query.spi.QueryParameterBindingValidator.validate(QueryParameterBindingValidator.java:54)
> at org.hibernate.query.spi.QueryParameterBindingValidator.validate(QueryParameterBindingValidator.java:27)
> at org.hibernate.query.internal.QueryParameterBindingImpl.validate(QueryParameterBindingImpl.java:90)
> at org.hibernate.query.internal.QueryParameterBindingImpl.setBindValue(QueryParameterBindingImpl.java:55)
> at org.hibernate.query.internal.AbstractProducedQuery.setParameter(AbstractProducedQuery.java:493)
> at org.hibernate.query.internal.AbstractProducedQuery.setParameter(AbstractProducedQuery.java:106)
> at org.hibernate.query.criteria.internal.compile.CriteriaCompiler$1$1.bind(CriteriaCompiler.java:119)
> at org.hibernate.query.criteria.internal.AbstractManipulationCriteriaQuery$1.buildCompiledQuery(AbstractManipulationCriteriaQuery.java:135)
> at org.hibernate.query.criteria.internal.compile.CriteriaCompiler.compile(CriteriaCompiler.java:149)
> at org.hibernate.internal.SessionImpl.createQuery(SessionImpl.java:3724)
> at org.hibernate.internal.SessionImpl.createQuery(SessionImpl.java:207)
> at org.infinispan.persistence.jpa.JpaStore.deleteBatch(JpaStore.java:341)
> ... 6 more
> If entries should be removed this is only done in memory but the store will fail ASYNC.
> Because of the async contract there is only a warning wihtout any exact hint:
> WARN: ISPN000053: Unable to process some async modifications after 10 retries!
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 11 months
[JBoss JIRA] (ISPN-10922) StateTransferLinkFailuresTest random failures
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10922?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10922:
--------------------------------
Status: Pull Request Sent (was: Open)
Git Pull Request: https://github.com/infinispan/infinispan/pull/7557
> StateTransferLinkFailuresTest random failures
> ---------------------------------------------
>
> Key: ISPN-10922
> URL: https://issues.jboss.org/browse/ISPN-10922
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 10.0.1.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.1.0.Beta1
>
>
> {{StateTransferLinkFailuresTest.testLinkBrokenDuringStateTransfer()}} tests that xsite state transfer eventually finishes when the link between sites is down. It simulates an exception in {{backupRemotely()}} and waits for 1 minute for state transfer to finish.
> The problem is that the default xsite state transfer retry configuration is to retry 30 times, waiting 2 seconds between retries. That means it's very possible the test will give up waiting on state transfer to finish just before state transfer finishes.
> {noformat}
> 21:43:02,524 DEBUG (transport-thread-Test-NodeA-p4484-t1:[]) [XSiteStateProviderImpl] [X-Site State Transfer - NYC-2] start DataContainer iteration
> 21:43:02,564 DEBUG (transport-thread-Test-NodeA-p4484-t1:[]) [XSiteStateProviderImpl] Sending chunk to site 'NYC-2'. Chunk contains [XSiteState{key=k_2, value=v_2, metadata=EmbeddedMetadata{version=null}}, XSiteState{key=k_4, value=v_4, metadata=EmbeddedMetadata{version=null}}]
> 21:44:02,579 TRACE (transport-thread-Test-NodeA-p4484-t1:[]) [RetryOnFailureXSiteCommand] Exception Response received. Exception is org.infinispan.util.concurrent.TimeoutException: induced timeout!
> 21:44:02,611 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.xsite.statetransfer.failures.StateTransferLinkFailuresTest.testLinkBrokenDuringStateTransfer[null, tx=false]
> java.lang.AssertionError:
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59) ~[testng-6.14.3.jar:?]
> at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24) ~[testng-6.14.3.jar:?]
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:250) ~[test-classes/:?]
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:232) ~[test-classes/:?]
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:208) ~[test-classes/:?]
> at org.infinispan.xsite.AbstractXSiteTest.assertEventuallyInSite(AbstractXSiteTest.java:193) ~[test-classes/:?]
> at org.infinispan.xsite.statetransfer.failures.StateTransferLinkFailuresTest.testLinkBrokenDuringStateTransfer(StateTransferLinkFailuresTest.java:89) ~[test-classes/:?]
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 11 months
[JBoss JIRA] (ISPN-10922) StateTransferLinkFailuresTest random failures
by Dan Berindei (Jira)
[ https://issues.jboss.org/browse/ISPN-10922?page=com.atlassian.jira.plugin... ]
Dan Berindei updated ISPN-10922:
--------------------------------
Status: Open (was: New)
> StateTransferLinkFailuresTest random failures
> ---------------------------------------------
>
> Key: ISPN-10922
> URL: https://issues.jboss.org/browse/ISPN-10922
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 10.0.1.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Labels: testsuite_stability
> Fix For: 10.1.0.Beta1
>
>
> {{StateTransferLinkFailuresTest.testLinkBrokenDuringStateTransfer()}} tests that xsite state transfer eventually finishes when the link between sites is down. It simulates an exception in {{backupRemotely()}} and waits for 1 minute for state transfer to finish.
> The problem is that the default xsite state transfer retry configuration is to retry 30 times, waiting 2 seconds between retries. That means it's very possible the test will give up waiting on state transfer to finish just before state transfer finishes.
> {noformat}
> 21:43:02,524 DEBUG (transport-thread-Test-NodeA-p4484-t1:[]) [XSiteStateProviderImpl] [X-Site State Transfer - NYC-2] start DataContainer iteration
> 21:43:02,564 DEBUG (transport-thread-Test-NodeA-p4484-t1:[]) [XSiteStateProviderImpl] Sending chunk to site 'NYC-2'. Chunk contains [XSiteState{key=k_2, value=v_2, metadata=EmbeddedMetadata{version=null}}, XSiteState{key=k_4, value=v_4, metadata=EmbeddedMetadata{version=null}}]
> 21:44:02,579 TRACE (transport-thread-Test-NodeA-p4484-t1:[]) [RetryOnFailureXSiteCommand] Exception Response received. Exception is org.infinispan.util.concurrent.TimeoutException: induced timeout!
> 21:44:02,611 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.xsite.statetransfer.failures.StateTransferLinkFailuresTest.testLinkBrokenDuringStateTransfer[null, tx=false]
> java.lang.AssertionError:
> at org.testng.AssertJUnit.fail(AssertJUnit.java:59) ~[testng-6.14.3.jar:?]
> at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24) ~[testng-6.14.3.jar:?]
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:250) ~[test-classes/:?]
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:232) ~[test-classes/:?]
> at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:208) ~[test-classes/:?]
> at org.infinispan.xsite.AbstractXSiteTest.assertEventuallyInSite(AbstractXSiteTest.java:193) ~[test-classes/:?]
> at org.infinispan.xsite.statetransfer.failures.StateTransferLinkFailuresTest.testLinkBrokenDuringStateTransfer(StateTransferLinkFailuresTest.java:89) ~[test-classes/:?]
> {noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 11 months
[JBoss JIRA] (ISPN-10922) StateTransferLinkFailuresTest random failures
by Dan Berindei (Jira)
Dan Berindei created ISPN-10922:
-----------------------------------
Summary: StateTransferLinkFailuresTest random failures
Key: ISPN-10922
URL: https://issues.jboss.org/browse/ISPN-10922
Project: Infinispan
Issue Type: Bug
Components: Test Suite - Core
Affects Versions: 10.0.1.Final
Reporter: Dan Berindei
Assignee: Dan Berindei
Fix For: 10.1.0.Beta1
{{StateTransferLinkFailuresTest.testLinkBrokenDuringStateTransfer()}} tests that xsite state transfer eventually finishes when the link between sites is down. It simulates an exception in {{backupRemotely()}} and waits for 1 minute for state transfer to finish.
The problem is that the default xsite state transfer retry configuration is to retry 30 times, waiting 2 seconds between retries. That means it's very possible the test will give up waiting on state transfer to finish just before state transfer finishes.
{noformat}
21:43:02,524 DEBUG (transport-thread-Test-NodeA-p4484-t1:[]) [XSiteStateProviderImpl] [X-Site State Transfer - NYC-2] start DataContainer iteration
21:43:02,564 DEBUG (transport-thread-Test-NodeA-p4484-t1:[]) [XSiteStateProviderImpl] Sending chunk to site 'NYC-2'. Chunk contains [XSiteState{key=k_2, value=v_2, metadata=EmbeddedMetadata{version=null}}, XSiteState{key=k_4, value=v_4, metadata=EmbeddedMetadata{version=null}}]
21:44:02,579 TRACE (transport-thread-Test-NodeA-p4484-t1:[]) [RetryOnFailureXSiteCommand] Exception Response received. Exception is org.infinispan.util.concurrent.TimeoutException: induced timeout!
21:44:02,611 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.xsite.statetransfer.failures.StateTransferLinkFailuresTest.testLinkBrokenDuringStateTransfer[null, tx=false]
java.lang.AssertionError:
at org.testng.AssertJUnit.fail(AssertJUnit.java:59) ~[testng-6.14.3.jar:?]
at org.testng.AssertJUnit.assertTrue(AssertJUnit.java:24) ~[testng-6.14.3.jar:?]
at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:250) ~[test-classes/:?]
at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:232) ~[test-classes/:?]
at org.infinispan.test.AbstractInfinispanTest.eventually(AbstractInfinispanTest.java:208) ~[test-classes/:?]
at org.infinispan.xsite.AbstractXSiteTest.assertEventuallyInSite(AbstractXSiteTest.java:193) ~[test-classes/:?]
at org.infinispan.xsite.statetransfer.failures.StateTransferLinkFailuresTest.testLinkBrokenDuringStateTransfer(StateTransferLinkFailuresTest.java:89) ~[test-classes/:?]
{noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
4 years, 11 months