[JBoss JIRA] (ISPN-3947) HotRod client keep trying recover connections to a failed cluster
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-3947?page=com.atlassian.jira.plugin.... ]
Dan Berindei commented on ISPN-3947:
------------------------------------
[~wfink] The thing is, we never retry on the same server. We retry on a random server regardless of where the key maps to, because it's better to hit the "wrong" server than to hit no server at all. It's true that picking a random server means it's also possible to get the same server every time, but that would be very unlikely.
This is similar to how {{numOwners}} is fixed in a distributed cache, regardless of the cluster size. Since {{numOwner}} server failures in a short interval of time would cause data loss anyway, throwing an exception should also be ok.
> HotRod client keep trying recover connections to a failed cluster
> -----------------------------------------------------------------
>
> Key: ISPN-3947
> URL: https://issues.jboss.org/browse/ISPN-3947
> Project: Infinispan
> Issue Type: Feature Request
> Components: Remote Protocols
> Affects Versions: 6.0.1.Final, 7.0.0.Alpha1
> Reporter: Wolf-Dieter Fink
> Assignee: Galder Zamarreño
> Labels: hotrod, hotrod-java-client
>
> If an infinispan-server cluster is not longer reachable for some reason, i.e. network disconnect, the hot-rod client try to re-establish the lost connections.
> The client library will retry this by a fixed calculation based on the max numbers of connections from the pool or 10 multiplied with the number of available servers.
> This can lead in a very long time until the application can continue and react as it will wait for the read- or connect-timeout for each try.
> To improve this behaviour there should be a configurable limit of retries per server and/or a timeout in total.
> This will give the application the chance to handle a remote-cache failure and reply to the user instead of hanging for minutes (with the default settings)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 3 months
[JBoss JIRA] (ISPN-3760) DistSyncL1FuncTest.testEntryInL1ReplaceWithConcurrentPut random failures
by William Burns (JIRA)
[ https://issues.jboss.org/browse/ISPN-3760?page=com.atlassian.jira.plugin.... ]
William Burns closed ISPN-3760.
-------------------------------
Resolution: Duplicate Issue
This appears to be the same as ISPN-3944.
> DistSyncL1FuncTest.testEntryInL1ReplaceWithConcurrentPut random failures
> ------------------------------------------------------------------------
>
> Key: ISPN-3760
> URL: https://issues.jboss.org/browse/ISPN-3760
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite - Core
> Affects Versions: 6.0.0.Final
> Reporter: Dan Berindei
> Assignee: William Burns
> Labels: testsuite_stability
> Fix For: 7.0.0.Final
>
>
> The test does a {{put(k, v)}} from an owner, then a {{get(k)}} from a non-owner, and assumes that {{k}} is now in the non-owner's L1 cache.
> But a previous test, {{testEntryInL1ReplaceWithConcurrentInvalidation}}, added the non-owner as a L1 requestor on the backup owner, which will try to invalidate the key on the non-owner. Because the L1 invalidation is asynchronous, it can reach the non-owner after the get was issued and invalidate the L1 entry:
> {noformat}
> 12:23:57,332 TRACE (testng-DistSyncL1FuncTest:dist) [InvocationContextInterceptor] Invoked with command PutKeyValueCommand{key=key-to-the-cache, value=first-put, flags=null, putIfAbsent=false, valueMatchingPolicy=MATCH_ALWAYS, metadata=EmbeddedMetadata{version=null}, successful=true} and InvocationContext [org.infinispan.context.SingleKeyNonTxInvocationContext@5caff826]
> 12:23:57,332 TRACE (testng-DistSyncL1FuncTest:dist) [JGroupsTransport] dests=[NodeD-17616, NodeC-12131], command=SingleRpcCommand{cacheName='dist', command=PutKeyValueCommand{key=key-to-the-cache, value=first-put, flags=null, putIfAbsent=false, valueMatchingPolicy=MATCH_ALWAYS, metadata=EmbeddedMetadata{version=null}, successful=true}}, mode=SYNCHRONOUS, timeout=60000
> 12:23:57,334 TRACE (asyncTransportThread-0,NodeC:) [JGroupsTransport] dests=[NodeA-23135], command=SingleRpcCommand{cacheName='dist', command=InvalidateL1Command{num keys=1, forRehash=false, origin=NodeD-17616}}, mode=SYNCHRONOUS, timeout=60000
> 12:23:57,339 TRACE (remote-thread-0,NodeC:dist) [DefaultDataContainer] Store ImmortalCacheEntry{key=key-to-the-cache, value=first-put} in container
> 12:23:57,339 TRACE (testng-DistSyncL1FuncTest:dist) [DefaultDataContainer] Store ImmortalCacheEntry{key=key-to-the-cache, value=first-put} in container
> 12:23:57,339 TRACE (testng-DistSyncL1FuncTest:dist) [L1ManagerImpl] Invalidating keys [key-to-the-cache] on nodes [NodeA-23135]. Use multicast? false
> 12:23:57,340 TRACE (testng-DistSyncL1FuncTest:dist) [InvocationContextInterceptor] Invoked with command GetKeyValueCommand {key=key-to-the-cache, flags=null} and InvocationContext [org.infinispan.context.SingleKeyNonTxInvocationContext@359e1aed]
> 12:23:57,340 TRACE (asyncTransportThread-4,NodeD:) [JGroupsTransport] dests=[NodeA-23135], command=SingleRpcCommand{cacheName='dist', command=InvalidateL1Command{num keys=1, forRehash=false, origin=null}}, mode=SYNCHRONOUS, timeout=60000
> 12:23:57,340 TRACE (testng-DistSyncL1FuncTest:dist) [JGroupsTransport] dests=[NodeD-17616, NodeC-12131], command=ClusteredGetCommand{key=key-to-the-cache, flags=null}, mode=WAIT_FOR_VALID_RESPONSE, timeout=60000
> 12:23:57,340 TRACE (remote-thread-1,NodeA:dist) [L1NonTxInterceptor] Aborted possible L1 update due to concurrent invalidation for key key-to-the-cache
> 12:23:57,341 ERROR (testng-DistSyncL1FuncTest:) [UnitTestTestNGListener] Test testEntryInL1ReplaceWithConcurrentPut(org.infinispan.distribution.DistSyncL1FuncTest) failed.
> java.lang.AssertionError: Entry for key [key-to-the-cache] should be in L1 on cache at [NodeA-23135]!
> at org.infinispan.distribution.DistributionTestHelper.assertIsInL1(DistributionTestHelper.java:31)
> at org.infinispan.distribution.BaseDistFunctionalTest.assertIsInL1(BaseDistFunctionalTest.java:183)
> at org.infinispan.distribution.DistSyncL1FuncTest.testEntryInL1ReplaceWithConcurrentPut(DistSyncL1FuncTest.java:180)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 3 months
[JBoss JIRA] (ISPN-3948) Missing logic to add the store to the configuration for custom stores
by Kurt Lehrke (JIRA)
[ https://issues.jboss.org/browse/ISPN-3948?page=com.atlassian.jira.plugin.... ]
Kurt Lehrke edited comment on ISPN-3948 at 1/30/14 8:55 AM:
------------------------------------------------------------
I did some research on possibly fixing this issue, here are some notes:
The PersistenceConfigurationBuilder is a bit misleading with the methods addStore. It doesn't add stores at all, it expects StoreConfigurationBuilders. I'm guessing the naming convention came along from the old version of the loaders configuration builder.
The xml defined configuration expects a class to the Store, not the StoreConfigurationBuilder. Unfortunately, there's no insight into what StoreConfigurationBuilder goes with what Cache Store/Loader. This makes it impossible to give the PersistenceConfigurationBuilder the means to build the store. My first reaction was to reuse the BuiltBy annotation for custom cachestores to specify which builder goes with it, that way it does have some means of getting the StoreConfigurationBuilder. The downfall to this approach is that even for the most simple custom cachestores (ones that might be self contained needing no additional configuration attributes/values), would still require a StoreConfiguration and a StoreConfigurationBuilder. This makes it especially more difficult because they can't be reused (say if there was a DefaultStoreConfiguration(Builder) )because of the annotations required by the StoreConfiguration (i.e. ConfigurationFor ).
The other option might be to allow custom cache stores to define the class to the StoreConfigurationBuilder. This would still require a builder for each store. Also, this would require a few changes in the Parser as the first step to the parsingStore method is reflectively creating the Store (even though it isn't used besides a null check). The upside to this approach is that it wouldn't require any additional annotation and the code would just have two paths for whether the configuration specified an actual store or a StoreConfigurationBuilder.
was (Author: kurtymckurt):
I did some research on possibly fixing this issue, here are some notes:
The PersistenceConfigurationBuilder is a bit misleading with the methods addStore. It doesn't add stores at all, it expects StoreConfigurationBuilders. I'm guessing the naming convention came along from the old version of the loaders configuration builder.
The xml defined configuration expects a class to the Store, not the StoreConfigurationBuilder. Unfortunately, there's no insight into what StoreConfigurationBuilder goes with what Cache Store/Loader. This makes it impossible to give the PersistenceConfigurationBuilder the means to build the store. My first reaction was to reuse the BuiltBy annotation for custom cachestores to specify which builder goes with it, that way it does have some means of getting the StoreConfigurationBuilder. The downfall to this approach is that even for the most simple custom cachestores (ones that might be self contained needing no additional configuration attributes/values), would still require a StoreConfiguration and a StoreConfigurationBuilder. This makes it especially more difficult because they can't be reused (say if there was a DefaultStoreConfiguration(Builder) because of the annotations required by the StoreConfiguration (i.e. ConfigurationFor ).
The other option might be to allow custom cache stores to define the class to the StoreConfigurationBuilder. This would still require a builder for each store. Also, this would require a few changes in the Parser as the first step to the parsingStore method is reflectively creating the Store (even though it isn't used besides a null check). The upside to this approach is that it wouldn't require any additional annotation and the code would just have two paths for whether the configuration specified an actual store or a StoreConfigurationBuilder.
> Missing logic to add the store to the configuration for custom stores
> ---------------------------------------------------------------------
>
> Key: ISPN-3948
> URL: https://issues.jboss.org/browse/ISPN-3948
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Loaders and Stores
> Affects Versions: 6.0.0.Final
> Reporter: Kurt Lehrke
> Assignee: Dan Berindei
> Priority: Critical
>
> Looking at the Parser60 class, if you specify a custom store that isn't a singleFileStore or a clusterLoader, the store doesn't get added to the configuration. I found this when I specified my own CacheLoader and it wasn't instantiated like expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 3 months
[JBoss JIRA] (ISPN-3948) Missing logic to add the store to the configuration for custom stores
by Kurt Lehrke (JIRA)
[ https://issues.jboss.org/browse/ISPN-3948?page=com.atlassian.jira.plugin.... ]
Kurt Lehrke edited comment on ISPN-3948 at 1/30/14 8:50 AM:
------------------------------------------------------------
I did some research on possibly fixing this issue, here are some notes:
The PersistenceConfigurationBuilder is a bit misleading with the methods addStore. It doesn't add stores at all, it expects StoreConfigurationBuilders. I'm guessing the naming convention came along from the old version of the loaders configuration builder.
The xml defined configuration expects a class to the Store, not the StoreConfigurationBuilder. Unfortunately, there's no insight into what StoreConfigurationBuilder goes with what Cache Store/Loader. This makes it impossible to give the PersistenceConfigurationBuilder the means to build the store. My first reaction was to reuse the BuiltBy annotation for custom cachestores to specify which builder goes with it, that way it does have some means of getting the StoreConfigurationBuilder. The downfall to this approach is that even for the most simple custom cachestores (ones that might be self contained needing no additional configuration attributes/values), would still require a StoreConfiguration and a StoreConfigurationBuilder. This makes it especially more difficult because they can't be reused (say if there was a DefaultStoreConfiguration(Builder) because of the annotations required by the StoreConfiguration (i.e. ConfigurationFor ).
The other option might be to allow custom cache stores to define the class to the StoreConfigurationBuilder. This would still require a builder for each store. Also, this would require a few changes in the Parser as the first step to the parsingStore method is reflectively creating the Store (even though it isn't used besides a null check). The upside to this approach is that it wouldn't require any additional annotation and the code would just have two paths for whether the configuration specified an actual store or a StoreConfigurationBuilder.
was (Author: kurtymckurt):
I did some research on possibly fixing this issue, here are some notes:
The PersistenceConfigurationBuilder is a bit misleading with the methods addStore. It doesn't add stores at all, it expects StoreConfigurationBuilders. I'm guessing the naming convention came along from the old version of the loaders configuration builder.
The xml defined configuration expects a class to the Store, not the StoreConfigurationBuilder. Unfortunately, there's no insight into what StoreConfigurationBuilder goes with what Cache Store/Loader. This makes it impossible to give the PersistenceConfigurationBuilder the means to build the store. My first reaction was to reuse the BuiltBy annotation for custom cachestores to specify which builder goes with it, that way it does have some means of getting the StoreConfigurationBuilder. The downfall to this approach is that even for the most simple custom cachestores (ones that might be self contained needing no additional configuration attributes/values), would still require a StoreConfiguration and a StoreConfigurationBuilder. This makes it especially more difficult because they can't be reused (say if there was a DefaultStoreConfiguration(Builder) because of the annotations required by the StoreConfiguration (i.e. ConfigurationFor ).
The other option might be to allow custom cache stores to define the class to the StoreConfigurationBuilder. This would still require a builder for each store. Also, this would require a few changes in the Parser as the first step to the parsingStore method is reflectively creating the Store (even though it isn't used besides a null check). The upside to this approach is that it wouldn't require any additional annotation and the code would just have to paths for whether the configuration specified an actual store or a StoreConfigurationBuilder.
> Missing logic to add the store to the configuration for custom stores
> ---------------------------------------------------------------------
>
> Key: ISPN-3948
> URL: https://issues.jboss.org/browse/ISPN-3948
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Loaders and Stores
> Affects Versions: 6.0.0.Final
> Reporter: Kurt Lehrke
> Assignee: Dan Berindei
> Priority: Critical
>
> Looking at the Parser60 class, if you specify a custom store that isn't a singleFileStore or a clusterLoader, the store doesn't get added to the configuration. I found this when I specified my own CacheLoader and it wasn't instantiated like expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 3 months
[JBoss JIRA] (ISPN-3948) Missing logic to add the store to the configuration for custom stores
by Kurt Lehrke (JIRA)
[ https://issues.jboss.org/browse/ISPN-3948?page=com.atlassian.jira.plugin.... ]
Kurt Lehrke commented on ISPN-3948:
-----------------------------------
I did some research on possibly fixing this issue, here are some notes:
The PersistenceConfigurationBuilder is a bit misleading with the methods addStore. It doesn't add stores at all, it expects StoreConfigurationBuilders. I'm guessing the naming convention came along from the old version of the loaders configuration builder.
The xml defined configuration expects a class to the Store, not the StoreConfigurationBuilder. Unfortunately, there's no insight into what StoreConfigurationBuilder goes with what Cache Store/Loader. This makes it impossible to give the PersistenceConfigurationBuilder the means to build the store. My first reaction was to reuse the BuiltBy annotation for custom cachestores to specify which builder goes with it, that way it does have some means of getting the StoreConfigurationBuilder. The downfall to this approach is that even for the most simple custom cachestores (ones that might be self contained needing no additional configuration attributes/values), would still require a StoreConfiguration and a StoreConfigurationBuilder. This makes it especially more difficult because they can't be reused (say if there was a DefaultStoreConfiguration(Builder) because of the annotations required by the StoreConfiguration (i.e. ConfigurationFor ).
The other option might be to allow custom cache stores to define the class to the StoreConfigurationBuilder. This would still require a builder for each store. Also, this would require a few changes in the Parser as the first step to the parsingStore method is reflectively creating the Store (even though it isn't used besides a null check). The upside to this approach is that it wouldn't require any additional annotation and the code would just have to paths for whether the configuration specified an actual store or a StoreConfigurationBuilder.
> Missing logic to add the store to the configuration for custom stores
> ---------------------------------------------------------------------
>
> Key: ISPN-3948
> URL: https://issues.jboss.org/browse/ISPN-3948
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Loaders and Stores
> Affects Versions: 6.0.0.Final
> Reporter: Kurt Lehrke
> Assignee: Dan Berindei
> Priority: Critical
>
> Looking at the Parser60 class, if you specify a custom store that isn't a singleFileStore or a clusterLoader, the store doesn't get added to the configuration. I found this when I specified my own CacheLoader and it wasn't instantiated like expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 3 months
[JBoss JIRA] (ISPN-3947) HotRod client keep trying recover connections to a failed cluster
by Wolf-Dieter Fink (JIRA)
[ https://issues.jboss.org/browse/ISPN-3947?page=com.atlassian.jira.plugin.... ]
Wolf-Dieter Fink commented on ISPN-3947:
----------------------------------------
Initially my idea was to have both,
max-per-server to prevent from retrying the same server x-times
max-in-total to stop after a number of retries in a big cluster
so retries = max-in-total > (max-per-server*noOfServers) ? (max-per-server*noOfServers) : max-in-total
In this case it is ensured that there is a threshold of retries but it's possible to try each server in a growing cluster.
But I think it will be difficult to set the retries independent from a bit environment knowlege ;)
> HotRod client keep trying recover connections to a failed cluster
> -----------------------------------------------------------------
>
> Key: ISPN-3947
> URL: https://issues.jboss.org/browse/ISPN-3947
> Project: Infinispan
> Issue Type: Feature Request
> Components: Remote Protocols
> Affects Versions: 6.0.1.Final, 7.0.0.Alpha1
> Reporter: Wolf-Dieter Fink
> Assignee: Galder Zamarreño
> Labels: hotrod, hotrod-java-client
>
> If an infinispan-server cluster is not longer reachable for some reason, i.e. network disconnect, the hot-rod client try to re-establish the lost connections.
> The client library will retry this by a fixed calculation based on the max numbers of connections from the pool or 10 multiplied with the number of available servers.
> This can lead in a very long time until the application can continue and react as it will wait for the read- or connect-timeout for each try.
> To improve this behaviour there should be a configurable limit of retries per server and/or a timeout in total.
> This will give the application the chance to handle a remote-cache failure and reply to the user instead of hanging for minutes (with the default settings)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 3 months
[JBoss JIRA] (ISPN-3957) Preload with async cache store is not efficient
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-3957?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-3957:
-----------------------------------------------
Dan Berindei <dberinde(a)redhat.com> changed the Status of [bug 1059489|https://bugzilla.redhat.com/show_bug.cgi?id=1059489] from NEW to MODIFIED
> Preload with async cache store is not efficient
> -----------------------------------------------
>
> Key: ISPN-3957
> URL: https://issues.jboss.org/browse/ISPN-3957
> Project: Infinispan
> Issue Type: Enhancement
> Components: Loaders and Stores
> Affects Versions: 5.2.7.Final
> Reporter: Mircea Markus
> Assignee: Dan Berindei
> Fix For: 5.2.8.Final, 7.0.0.Alpha1, 7.0.0.Final
>
>
> Configuring on a AdvancedCacheLoader preload=true and asyn=true cause it to load each entry in the store in a loop, each entry being loaded through an store read.
> This is caused by the way loadAll is implemented in the AsynLoader: in order to enforce consistency with whatever is in memory it does some special handling. The thing is, though, that we don't need this advanced async loader logic during the initial preload, as the async cache loader is empty.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 3 months
[JBoss JIRA] (ISPN-3957) Preload with async cache store is not efficient
by Dan Berindei (JIRA)
[ https://issues.jboss.org/browse/ISPN-3957?page=com.atlassian.jira.plugin.... ]
Dan Berindei updated ISPN-3957:
-------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
> Preload with async cache store is not efficient
> -----------------------------------------------
>
> Key: ISPN-3957
> URL: https://issues.jboss.org/browse/ISPN-3957
> Project: Infinispan
> Issue Type: Enhancement
> Components: Loaders and Stores
> Affects Versions: 5.2.7.Final
> Reporter: Mircea Markus
> Assignee: Dan Berindei
> Fix For: 5.2.8.Final, 7.0.0.Alpha1, 7.0.0.Final
>
>
> Configuring on a AdvancedCacheLoader preload=true and asyn=true cause it to load each entry in the store in a loop, each entry being loaded through an store read.
> This is caused by the way loadAll is implemented in the AsynLoader: in order to enforce consistency with whatever is in memory it does some special handling. The thing is, though, that we don't need this advanced async loader logic during the initial preload, as the async cache loader is empty.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
10 years, 3 months