[JBoss JIRA] (ISPN-7070) CacheNotifierImplInitialTransferDistTest.testIterationBeganAndSegmentNotComplete random failures
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-7070?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-7070:
----------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 9.2.0.Beta2
(was: 9.0.0.CR4)
Resolution: Done
> CacheNotifierImplInitialTransferDistTest.testIterationBeganAndSegmentNotComplete random failures
> ------------------------------------------------------------------------------------------------
>
> Key: ISPN-7070
> URL: https://issues.jboss.org/browse/ISPN-7070
> Project: Infinispan
> Issue Type: Bug
> Components: Core, Test Suite - Core
> Affects Versions: 9.0.0.Alpha4
> Reporter: Dan Berindei
> Assignee: William Burns
> Priority: Critical
> Labels: testsuite_stability
> Fix For: 9.2.0.Beta2
>
> Attachments: CacheNotifierImplInitialTransferDistTest_ISPN-5467_CompletableFuture-like_API_20161003.log, CacheNotifierImplInitialTransferDistTest_ISPN-7919_RpcManager_ResponseCollector_20171107.log.gz
>
>
> Looks like sometimes the originator listener ignores the CACHE_ENTRY_REMOVED notification:
> {noformat}
> 09:58:57,877 INFO (testng-Test:[]) [TestSuiteProgress] Test starting: org.infinispan.notifications.cachelistener.CacheNotifierImplInitialTransferDistTest.testRemoveAfterIterationBeganAndSegmentNotCompleteValueNonOwnerClustered
> 09:58:58,913 DEBUG (testng-Test:[]) [Test] Found key key-to-change-0 with primary owner != Test-NodeA-7051, segment 60
> 09:58:58,930 TRACE (remote-thread-Test-NodeA-p23726-t6:[]) [ReadCommittedEntry] Updating entry (key=key-to-change-0 removed=false valid=true changed=true created=true value=initial-value metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, providedMetadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null})
> 09:58:58,972 TRACE (remote-thread-Test-NodeC-p23887-t6:[]) [ReadCommittedEntry] Updating entry (key=key-to-change-0 removed=false valid=true changed=true created=true value=initial-value metadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null}, providedMetadata=EmbeddedExpirableMetadata{lifespan=-1, maxIdle=-1, version=null})
> 09:58:59,014 TRACE (ForkThread-1,Test:[]) [Test] Adding segment listener 2a41ec3f-c79f-4b7d-bd0e-8754170d8fc1 : org.infinispan.notifications.cachelistener.DistributedQueueingSegmentListener@68007f8
> 09:58:59,014 TRACE (ForkThread-1,Test:[]) [CacheNotifierImpl] Replicating cluster listener to other nodes [Test-NodeA-7051, Test-NodeB-10318, Test-NodeC-52778] for cluster listener with id 2a41ec3f-c79f-4b7d-bd0e-8754170d8fc1
> 09:58:58,998 TRACE (testng-Test:[]) [CheckPoint] Waiting for event pre_complete_segment_invoked * 1
> 09:58:59,063 TRACE (remote-thread-Test-NodeC-p23887-t6:[]) [ClusterListenerReplicateCallable] Registered local cluster listener for remote cluster listener from origin Test-NodeA-7051 with id 2a41ec3f-c79f-4b7d-bd0e-8754170d8fc1
> 09:58:59,098 TRACE (remote-thread-Test-NodeB-p23775-t6:[]) [ClusterListenerReplicateCallable] Registered local cluster listener for remote cluster listener from origin Test-NodeA-7051 with id 2a41ec3f-c79f-4b7d-bd0e-8754170d8fc1
> 09:58:59,145 TRACE (ForkThread-1,Test:[]) [CacheNotifierImpl] Listener 2a41ec3f-c79f-4b7d-bd0e-8754170d8fc1 requests initial state for cache
> 09:58:59,373 TRACE (ForkThread-1,Test:[]) [Test] Completed segments [48, 2, 229, 118, 71, 72, 233, 60]
> 09:58:59,373 TRACE (ForkThread-1,Test:[]) [Test] Listener received event EventImpl{type=CACHE_ENTRY_CREATED, pre=false, cache=Cache 'DistInitialTransferListener'@Test-NodeA-7051, key=key-to-change-0, value=initial-value, oldValue=null, transaction=null, originLocal=true, transactionSuccessful=false, entries=null, created=false}
> 09:58:59,373 TRACE (ForkThread-1,Test:[]) [Test] Notified for key key-to-change-0
> 09:58:59,373 TRACE (ForkThread-1,Test:[]) [CheckPoint] Triggering event pre_complete_segment_invoked * 1 (available = 1, total = 1)
> 09:58:59,373 TRACE (ForkThread-1,Test:[]) [CheckPoint] Waiting for event pre_complete_segment_released * 1
> 09:58:59,373 TRACE (testng-Test:[]) [CheckPoint] Received event pre_complete_segment_invoked * 1 (available = 0, total = 1)
> 09:58:59,577 TRACE (remote-thread-Test-NodeA-p23726-t6:[]) [ReadCommittedEntry] Updating entry (key=key-to-change-0 removed=true valid=false changed=true created=false value=initial-value metadata=EmbeddedMetadata{version=null}, providedMetadata=null)
> 09:58:59,661 TRACE (remote-thread-Test-NodeC-p23887-t5:[]) [ReadCommittedEntry] Updating entry (key=key-to-change-0 removed=true valid=false changed=true created=false value=initial-value metadata=EmbeddedMetadata{version=null}, providedMetadata=null)
> 09:58:59,661 TRACE (remote-thread-Test-NodeC-p23887-t5:[]) [RequestCorrelator] Test-NodeC-52778: invoking unicast RPC [req-id=91126] on Test-NodeA-7051
> 09:58:59,684 TRACE (OOB-2,Test-NodeA-7051:[]) [RequestCorrelator] calling (org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher) with request 91126
> 09:58:59,779 TRACE (remote-thread-Test-NodeA-p23726-t6:[]) [NonTotalOrderPerCacheInboundInvocationHandler] Calling perform() on DistributedExecuteCommand [cache=Cache 'DistInitialTransferListener'@Test-NodeA-7051, keys=[], callable=org.infinispan.notifications.cachelistener.cluster.ClusterEventCallable@640b0234]
> 09:58:59,780 TRACE (remote-thread-Test-NodeA-p23726-t6:[]) [ClusterEventCallable] Received cluster event(s) [ClusterEvent {type=CACHE_ENTRY_REMOVED, cache=Cache 'DistInitialTransferListener'@Test-NodeA-7051, key=key-to-change-0, value=null, oldValue=initial-value, transaction=null, retryCommand=false, origin=Test-NodeC-52778}], notifying cluster listener with id 2a41ec3f-c79f-4b7d-bd0e-8754170d8fc1
> 09:58:59,837 TRACE (testng-Test:[]) [CheckPoint] Triggering event pre_complete_segment_released * 999999999 (available = 999999999, total = 999999999)
> 09:58:59,837 TRACE (ForkThread-1,Test:[]) [CheckPoint] Received event pre_complete_segment_released * 1 (available = 999999998, total = 999999999)
> 09:58:59,898 TRACE (ForkThread-1,Test:[]) [CacheNotifierImpl] Listener 2a41ec3f-c79f-4b7d-bd0e-8754170d8fc1 initial state for cache completed
> 09:58:59,916 TRACE (remote-thread-Test-NodeB-p23775-t6:[]) [ClusterListenerRemoveCallable] Removing local cluster listener due to parent cluster listener was removed : 2a41ec3f-c79f-4b7d-bd0e-8754170d8fc1
> 09:58:59,921 TRACE (remote-thread-Test-NodeC-p23887-t5:[]) [ClusterListenerRemoveCallable] Removing local cluster listener due to parent cluster listener was removed : 2a41ec3f-c79f-4b7d-bd0e-8754170d8fc1
> 09:58:59,943 ERROR (testng-Test:[]) [TestSuiteProgress] Test failed: org.infinispan.notifications.cachelistener.CacheNotifierImplInitialTransferDistTest.testRemoveAfterIterationBeganAndSegmentNotCompleteValueNonOwnerClustered
> java.lang.AssertionError: expected [12] but found [11]
> {noformat}
> {{CacheNotifierImplInitialTransferDistTest}} also had a different problem: it replaced the {{CacheNotifier}} component with a clone before adding the listener, but replaced it again with the original before removing the listener. Because the listener's uuid was only registered in the clone, {{removeListener}} couldn't find the uuid and disn't try to remove the listener on the other nodes. This meant the listener was leaked, resulting in logs like this:
> {noformat}
> 20:12:04,098 TRACE (remote-thread-Test-NodeB-p24127-t6:[]) [ReadCommittedEntry] Updating entry (key=key-to-change-0 removed=true valid=false changed=true created=false value=initial-value metadata=EmbeddedMetadata{version=null}, providedMetadata=null)
> 20:12:04,099 TRACE (remote-thread-Test-NodeB-p24127-t6:[]) [RemoteClusterListener] Passing Event to manager EventImpl{type=CACHE_ENTRY_REMOVED, pre=false, cache=Cache 'DistInitialTransferListener'@Test-NodeB-56142, key=key-to-change-0, value=null, oldValue=initial-value, transaction=null, originLocal=false, transactionSuccessful=false, entries=null, created=false} to send to Test-NodeA-38047
> 20:12:04,100 TRACE (remote-thread-Test-NodeB-p24127-t6:[]) [RemoteClusterListener] Passing Event to manager EventImpl{type=CACHE_ENTRY_REMOVED, pre=false, cache=Cache 'DistInitialTransferListener'@Test-NodeB-56142, key=key-to-change-0, value=null, oldValue=initial-value, transaction=null, originLocal=false, transactionSuccessful=false, entries=null, created=false} to send to Test-NodeA-38047
> 20:12:04,100 TRACE (remote-thread-Test-NodeB-p24127-t6:[]) [RemoteClusterListener] Passing Event to manager EventImpl{type=CACHE_ENTRY_REMOVED, pre=false, cache=Cache 'DistInitialTransferListener'@Test-NodeB-56142, key=key-to-change-0, value=null, oldValue=initial-value, transaction=null, originLocal=false, transactionSuccessful=false, entries=null, created=false} to send to Test-NodeA-38047
> 20:12:04,100 TRACE (remote-thread-Test-NodeB-p24127-t6:[]) [RemoteClusterListener] Passing Event to manager EventImpl{type=CACHE_ENTRY_REMOVED, pre=false, cache=Cache 'DistInitialTransferListener'@Test-NodeB-56142, key=key-to-change-0, value=null, oldValue=initial-value, transaction=null, originLocal=false, transactionSuccessful=false, entries=null, created=false} to send to Test-NodeA-38047
> 20:12:04,100 TRACE (remote-thread-Test-NodeB-p24127-t6:[]) [RemoteClusterListener] Passing Event to manager EventImpl{type=CACHE_ENTRY_REMOVED, pre=false, cache=Cache 'DistInitialTransferListener'@Test-NodeB-56142, key=key-to-change-0, value=null, oldValue=initial-value, transaction=null, originLocal=false, transactionSuccessful=false, entries=null, created=false} to send to Test-NodeA-38047
> 20:12:04,106 TRACE (remote-thread-Test-NodeA-p24103-t6:[]) [MultiClusterEventCallable] Received multiple cluster event(s) {86f77630-b1bb-4a21-97bb-b03efeae39d6=[ClusterEvent {type=CACHE_ENTRY_REMOVED, cache=Cache 'DistInitialTransferListener'@Test-NodeA-38047, key=key-to-change-0, value=null, oldValue=initial-value, transaction=null, retryCommand=false, origin=Test-NodeB-56142}], 91d05a50-4a35-4e37-8c5c-bffbabc6154b=[ClusterEvent {type=CACHE_ENTRY_REMOVED, cache=Cache 'DistInitialTransferListener'@Test-NodeA-38047, key=key-to-change-0, value=null, oldValue=initial-value, transaction=null, retryCommand=false, origin=Test-NodeB-56142}], ba09480f-c698-431b-8fe6-213f1bdefc7b=[ClusterEvent {type=CACHE_ENTRY_REMOVED, cache=Cache 'DistInitialTransferListener'@Test-NodeA-38047, key=key-to-change-0, value=null, oldValue=initial-value, transaction=null, retryCommand=false, origin=Test-NodeB-56142}], 992da014-0516-4f2a-8093-50ba3b1f358a=[ClusterEvent {type=CACHE_ENTRY_REMOVED, cache=Cache 'DistInitialTransferListener'@Test-NodeA-38047, key=key-to-change-0, value=null, oldValue=initial-value, transaction=null, retryCommand=false, origin=Test-NodeB-56142}], b4eeb9b3-85c4-4ff9-865e-076b7cf93faa=[ClusterEvent {type=CACHE_ENTRY_REMOVED, cache=Cache 'DistInitialTransferListener'@Test-NodeA-38047, key=key-to-change-0, value=null, oldValue=initial-value, transaction=null, retryCommand=false, origin=Test-NodeB-56142}]}
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8565) Caches are not able to recover from ISPN000136: Error executing command SizeCommand
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-8565?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant reassigned ISPN-8565:
-------------------------------------
Assignee: Dan Berindei
> Caches are not able to recover from ISPN000136: Error executing command SizeCommand
> -----------------------------------------------------------------------------------
>
> Key: ISPN-8565
> URL: https://issues.jboss.org/browse/ISPN-8565
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.1.3.Final
> Environment: 8 ARM based nodes with running Debian Linux and Oracle Java.
> Reporter: Michal Stehlik
> Assignee: Dan Berindei
> Attachments: infinity_configuration.zip, nodes_infinispan_logs.zip
>
>
> During parallel connecting nodes to cloud, exception appears in Infinispan logs. Then any query to any cache waits infinite time to complete without response. In attachments is present configuration of Infinispan & JGroups ( all nodes has same config) and infinity logs from all nodes. Trace logs was provided for following classes: org.infinispan.statetransfer.StateConsumerImpl, org.infinispan.statetransfer.StateProviderImpl, org.infinispan.remoting.rpc.RpcManagerImpl, org.infinispan.remoting.inboundhandler.NonTotalOrderTxPerCacheInboundInvocationHandler
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8566) Rest server doesn't handle wildcards in the Accept header
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8566?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-8566:
------------------------------------
Description:
Typically a browser automatically adds multiple mime types in the accept header, like:
"text/html, application.xhtml+xml,\*/\*"
But when requesting a document in a cache encoded with protostream with the headers above, the response is
"Cannot find transcoder between application/xml and application/x-protostream"
was:
Typically a browser automatically adds multiple mime types in the accept header, like:
"text/html, application.xhtml+xml,\*/\*"
But when requesting a document in a cache encoded with protostream, the response is
"Cannot find transcoder between application/xml and application/x-protostream"
> Rest server doesn't handle wildcards in the Accept header
> ---------------------------------------------------------
>
> Key: ISPN-8566
> URL: https://issues.jboss.org/browse/ISPN-8566
> Project: Infinispan
> Issue Type: Bug
> Reporter: Gustavo Fernandes
> Assignee: Gustavo Fernandes
>
> Typically a browser automatically adds multiple mime types in the accept header, like:
> "text/html, application.xhtml+xml,\*/\*"
> But when requesting a document in a cache encoded with protostream with the headers above, the response is
> "Cannot find transcoder between application/xml and application/x-protostream"
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8566) Rest server doesn't handle wildcards in the Accept header
by Gustavo Fernandes (JIRA)
Gustavo Fernandes created ISPN-8566:
---------------------------------------
Summary: Rest server doesn't handle wildcards in the Accept header
Key: ISPN-8566
URL: https://issues.jboss.org/browse/ISPN-8566
Project: Infinispan
Issue Type: Bug
Reporter: Gustavo Fernandes
Typically a browser automatically adds multiple mime types in the accept header, like:
"text/html, application.xhtml+xml,*/*"
But when requesting a document in a cache encoded with protostream, the response is
"Cannot find transcoder between application/xml and application/x-protostream"
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years
[JBoss JIRA] (ISPN-8566) Rest server doesn't handle wildcards in the Accept header
by Gustavo Fernandes (JIRA)
[ https://issues.jboss.org/browse/ISPN-8566?page=com.atlassian.jira.plugin.... ]
Gustavo Fernandes updated ISPN-8566:
------------------------------------
Description:
Typically a browser automatically adds multiple mime types in the accept header, like:
"text/html, application.xhtml+xml,\*/\*"
But when requesting a document in a cache encoded with protostream, the response is
"Cannot find transcoder between application/xml and application/x-protostream"
was:
Typically a browser automatically adds multiple mime types in the accept header, like:
"text/html, application.xhtml+xml,*/*"
But when requesting a document in a cache encoded with protostream, the response is
"Cannot find transcoder between application/xml and application/x-protostream"
> Rest server doesn't handle wildcards in the Accept header
> ---------------------------------------------------------
>
> Key: ISPN-8566
> URL: https://issues.jboss.org/browse/ISPN-8566
> Project: Infinispan
> Issue Type: Bug
> Reporter: Gustavo Fernandes
>
> Typically a browser automatically adds multiple mime types in the accept header, like:
> "text/html, application.xhtml+xml,\*/\*"
> But when requesting a document in a cache encoded with protostream, the response is
> "Cannot find transcoder between application/xml and application/x-protostream"
--
This message was sent by Atlassian JIRA
(v7.5.0#75005)
8 years