[JBoss JIRA] (ISPN-11687) Infinispan not able to start after CR configuration change
by Ricardo Zanini Fernandes (Jira)
[ https://issues.redhat.com/browse/ISPN-11687?page=com.atlassian.jira.plugi... ]
Ricardo Zanini Fernandes commented on ISPN-11687:
-------------------------------------------------
Hi guys!
Feel free to contact me if you need further assistance configuring the environment/using the Kogito Operator.
> Infinispan not able to start after CR configuration change
> ----------------------------------------------------------
>
> Key: ISPN-11687
> URL: https://issues.redhat.com/browse/ISPN-11687
> Project: Infinispan
> Issue Type: Bug
> Components: OpenShift, Operator
> Affects Versions: 10.1.2.Final
> Reporter: Marian Macik
> Assignee: Vittorio Rigamonti
> Priority: Major
>
> Hi guys, as you may know, Kogito Operator uses Infinispan Operator to create a predefined Infinispan instance running on Openshift. Quite often we encounter an issue reproducible with these steps:
> 1. Install the Kogito Operator (this will also install Infinispan Operator).
> 2. Create a KogitoApp custom resource (CR) with this YAML:
> {code:yaml}
> apiVersion: app.kiegroup.org/v1alpha1
> kind: KogitoApp
> metadata:
> name: example-quarkus
> spec:
> enablePersistence: true
> build:
> envs:
> # enable persistence
> - name: MAVEN_ARGS_APPEND
> value: "-Ppersistence"
> gitSource:
> contextDir: process-quarkus-example
> uri: 'https://github.com/kiegroup/kogito-examples'
> reference: master
> {code}
> This will create a KogitoApp CR and will tell the Kogito Operator to provision Infinispan with one replica. Kogito application runs on Quarkus which makes use of RemoteCacheManager of Quarkus Infinispan Client Extension. Up to this point, everything works, application is deployed.
> 3. Try change Infinispan config by editing Infinispan CR a few times and Infinispan won't be able to start properly. By editing I mean - change one of these 3 parameters:
> {code:yaml}
> ...
> spec:
> container:
> cpu: ''
> extraJvmOpts: ''
> memory: ''
> ...
> {code}
> I generally want to change the cpu and memory as defaults are too low and I am also specifying `-Xmx2G` to extraJvmOpts so Infinispan has more heap than 200 MB which is default.
> Anyway, if you do this change a couple of times and after each change you wait until Infinispan pod is restarted, after ~5 times you will see [java.nio.channels.OverlappingFileLockException|https://gist.github.com/Ma...] in the Infinispan pod log.
> There is also another issue attached at the bottom of the Gist which was observed in Openshift events logs.
> What I have found is that if I create only KogitoInfra CR, which will create Infinispan CR and won't run any KogitoApp, so there is nothing connected to Infinispan, I can restart it how many times I want and it will work without any issues. I even tried to store something to the Infinispan via Infinispan REST API from the pod and tried changing Infinispan configuration then, and it worked like a charm after each restart.
> However, as soon as I deploy KogitoApp so it is connected to Infinispan using HotRod client and change Infinispan CR a few times (after each change waiting for Infinispan pod to restart), it will break with the linked exception present in the logs.
> To me it seems that this stops working once there is actual connection to Infinispan using HotRod client. I am not sure how this client works internally, but I would think that in addition to real user data there is some sort of exchange of "control data" let's say in this protocol between client and Infinispan which might break if Infinispan is suddenly restarted? Not sure, but with pushing there data using REST API (so without HotRod client) where the connection is maintained only for the time of the request, the exception didn't occur.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (ISPN-11693) Remote JCacheManager doesn't apply configuration from URI
by Tristan Tarrant (Jira)
Tristan Tarrant created ISPN-11693:
--------------------------------------
Summary: Remote JCacheManager doesn't apply configuration from URI
Key: ISPN-11693
URL: https://issues.redhat.com/browse/ISPN-11693
Project: Infinispan
Issue Type: Bug
Components: Remote Protocols
Affects Versions: 10.1.6.Final
Reporter: Nathan Mittlestat
Assignee: Tristan Tarrant
Fix For: 10.1.7.Final, 11.0.0.Dev05
Our project is using JCache API's backed by Infinspan. This is a client-server scenario where we are configuring the Hot Rod Java Client to connect to an Infinispan server.
We are using _javax.cache.spi.CachingProvider.getCacheManager(URI, ClassLoader, Properties)_ to obtain a _CacheManager_. The URI passed points to an infinispan XML that specifies replicated-cache-configuration as part of the configuration. However the cache created from _CacheManager.createCache()_ ends up being a local cache. This proves to be a problem when connecting to a cluster of Infinispan servers, as when other connections attempt to utilize the cached data, they connect to a different Infinispn sever that doesn't have the cached data. We have verified the XML file pointed to by the URI is on the ClassLoader passed to the _CacheManager_, and still see the same results.
When we attempt this same scenario using embedded mode instead of client server mode, the configuration in the URI is applied correctly. We have even been able to create a cluster of embedded Infinispan servers via JGroups and confirmed the caches are replicated and not local.
We have also attempted this scenario using Infinispan's proprietary APIs:
_RemoteCacheManager.administration().getOrCreateCache(String name, BasicConfiguration configuration)_
The configuration object passed to _RemoteCacheManagerAdmin.getOrCreateCache()_ loads the same Infinispan XML configuration used in the failing remote JCache scenario (as a _XMlStringConfiguration_ object).
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (ISPN-11627) InfinispanServerRule returning FileAlreadyExistsException if you use mavenArtifacts and restart the server
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-11627?page=com.atlassian.jira.plugi... ]
Tristan Tarrant updated ISPN-11627:
-----------------------------------
Fix Version/s: 10.1.7.Final
> InfinispanServerRule returning FileAlreadyExistsException if you use mavenArtifacts and restart the server
> ----------------------------------------------------------------------------------------------------------
>
> Key: ISPN-11627
> URL: https://issues.redhat.com/browse/ISPN-11627
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite
> Affects Versions: 11.0.0.Dev03
> Reporter: Gustavo Lira e Silva
> Assignee: Gustavo Lira e Silva
> Priority: Major
> Fix For: 10.1.7.Final, 11.0.0.Dev05
>
>
> If we use {{InfinispanServerRule}} deploying maven artifacts and restart the server during the tests, we will receive {{FileAlreadyExistsException}}
> {code:java}
> @ClassRule
> public static InfinispanServerRule SERVERS =
> InfinispanServerRuleBuilder.config("configuration/PersistenceTest.xml")
> .numServers(1)
> .runMode(ServerRunMode.CONTAINER)
> .mavenArtifacts("com.h2database:h2:1.4.199")
> .build();
> ...
> ContainerInfinispanServerDriver serverDriver = (ContainerInfinispanServerDriver) SERVER.getServerDriver();
> RestClient rest = SERVER_TEST.rest().get();
> sync(rest.server().stop());
> Eventually.eventually(
> "Cluster did not shutdown within timeout",
> () -> (!serverDriver.isRunning(0) ),
> serverDriver.getTimeout(), 1, TimeUnit.SECONDS);
> serverDriver.restart(0);
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (ISPN-11627) InfinispanServerRule returning FileAlreadyExistsException if you use mavenArtifacts and restart the server
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-11627?page=com.atlassian.jira.plugi... ]
Tristan Tarrant updated ISPN-11627:
-----------------------------------
Status: Open (was: New)
> InfinispanServerRule returning FileAlreadyExistsException if you use mavenArtifacts and restart the server
> ----------------------------------------------------------------------------------------------------------
>
> Key: ISPN-11627
> URL: https://issues.redhat.com/browse/ISPN-11627
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite
> Affects Versions: 11.0.0.Dev03
> Reporter: Gustavo Lira e Silva
> Assignee: Gustavo Lira e Silva
> Priority: Major
> Fix For: 10.1.7.Final, 11.0.0.Dev05
>
>
> If we use {{InfinispanServerRule}} deploying maven artifacts and restart the server during the tests, we will receive {{FileAlreadyExistsException}}
> {code:java}
> @ClassRule
> public static InfinispanServerRule SERVERS =
> InfinispanServerRuleBuilder.config("configuration/PersistenceTest.xml")
> .numServers(1)
> .runMode(ServerRunMode.CONTAINER)
> .mavenArtifacts("com.h2database:h2:1.4.199")
> .build();
> ...
> ContainerInfinispanServerDriver serverDriver = (ContainerInfinispanServerDriver) SERVER.getServerDriver();
> RestClient rest = SERVER_TEST.rest().get();
> sync(rest.server().stop());
> Eventually.eventually(
> "Cluster did not shutdown within timeout",
> () -> (!serverDriver.isRunning(0) ),
> serverDriver.getTimeout(), 1, TimeUnit.SECONDS);
> serverDriver.restart(0);
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (ISPN-11627) InfinispanServerRule returning FileAlreadyExistsException if you use mavenArtifacts and restart the server
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-11627?page=com.atlassian.jira.plugi... ]
Tristan Tarrant updated ISPN-11627:
-----------------------------------
Status: Resolved (was: Pull Request Sent)
Fix Version/s: 11.0.0.Dev05
Resolution: Done
> InfinispanServerRule returning FileAlreadyExistsException if you use mavenArtifacts and restart the server
> ----------------------------------------------------------------------------------------------------------
>
> Key: ISPN-11627
> URL: https://issues.redhat.com/browse/ISPN-11627
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite
> Affects Versions: 11.0.0.Dev03
> Reporter: Gustavo Lira e Silva
> Assignee: Gustavo Lira e Silva
> Priority: Major
> Fix For: 11.0.0.Dev05
>
>
> If we use {{InfinispanServerRule}} deploying maven artifacts and restart the server during the tests, we will receive {{FileAlreadyExistsException}}
> {code:java}
> @ClassRule
> public static InfinispanServerRule SERVERS =
> InfinispanServerRuleBuilder.config("configuration/PersistenceTest.xml")
> .numServers(1)
> .runMode(ServerRunMode.CONTAINER)
> .mavenArtifacts("com.h2database:h2:1.4.199")
> .build();
> ...
> ContainerInfinispanServerDriver serverDriver = (ContainerInfinispanServerDriver) SERVER.getServerDriver();
> RestClient rest = SERVER_TEST.rest().get();
> sync(rest.server().stop());
> Eventually.eventually(
> "Cluster did not shutdown within timeout",
> () -> (!serverDriver.isRunning(0) ),
> serverDriver.getTimeout(), 1, TimeUnit.SECONDS);
> serverDriver.restart(0);
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (ISPN-11627) InfinispanServerRule returning FileAlreadyExistsException if you use mavenArtifacts and restart the server
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-11627?page=com.atlassian.jira.plugi... ]
Tristan Tarrant updated ISPN-11627:
-----------------------------------
Status: Pull Request Sent (was: Open)
> InfinispanServerRule returning FileAlreadyExistsException if you use mavenArtifacts and restart the server
> ----------------------------------------------------------------------------------------------------------
>
> Key: ISPN-11627
> URL: https://issues.redhat.com/browse/ISPN-11627
> Project: Infinispan
> Issue Type: Bug
> Components: Test Suite
> Affects Versions: 11.0.0.Dev03
> Reporter: Gustavo Lira e Silva
> Assignee: Gustavo Lira e Silva
> Priority: Major
> Fix For: 10.1.7.Final, 11.0.0.Dev05
>
>
> If we use {{InfinispanServerRule}} deploying maven artifacts and restart the server during the tests, we will receive {{FileAlreadyExistsException}}
> {code:java}
> @ClassRule
> public static InfinispanServerRule SERVERS =
> InfinispanServerRuleBuilder.config("configuration/PersistenceTest.xml")
> .numServers(1)
> .runMode(ServerRunMode.CONTAINER)
> .mavenArtifacts("com.h2database:h2:1.4.199")
> .build();
> ...
> ContainerInfinispanServerDriver serverDriver = (ContainerInfinispanServerDriver) SERVER.getServerDriver();
> RestClient rest = SERVER_TEST.rest().get();
> sync(rest.server().stop());
> Eventually.eventually(
> "Cluster did not shutdown within timeout",
> () -> (!serverDriver.isRunning(0) ),
> serverDriver.getTimeout(), 1, TimeUnit.SECONDS);
> serverDriver.restart(0);
> {code}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (ISPN-11679) Bad uneven segment distributions can happen after the rebalancing with SyncConsistentHashFactory
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-11679?page=com.atlassian.jira.plugi... ]
Tristan Tarrant updated ISPN-11679:
-----------------------------------
Status: Open (was: New)
> Bad uneven segment distributions can happen after the rebalancing with SyncConsistentHashFactory
> ------------------------------------------------------------------------------------------------
>
> Key: ISPN-11679
> URL: https://issues.redhat.com/browse/ISPN-11679
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.4.19.Final, 10.1.6.Final
> Reporter: Masafumi Miura
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 11.0.0.Final
>
>
> Uneven segment distribution can happen after rebalancing with SyncConsistentHashFactory.
> Even if a larger number of segments than "20 * number of nodes" is configured, some nodes have only around 50-60% number of segments than other nodes after the rebalance in the worst case.
> I understand the documentation (like the following) states about the possibility of slight uneven distribution as a potential drawback of SyncConsistentHashFactory.
> [Infinispan 10 configuration guide|https://infinispan.org/docs/stable/titles/configuring/configuring.h...] says:
> {quote}
> This implementation does have some negative points in that the load distribution is slightly uneven. It also moves more segments than strictly necessary on a join or leave.
> {quote}
> [JDG 7 Developer Guide|https://access.redhat.com/documentation/en-us/red_hat_data_grid/7.2...] says:
> {quote}
> It has a potential drawback in that it can move a greater number of segments than necessary during re-balancing. This can be mitigated by using a larger number of segments.
> Another potential drawback is that the segments are not distributed as evenly as possible, and actually using a very large number of segments can make the distribution of segments worse.
> {quote}
> However, only half segments than others is not a slight uneven.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (ISPN-11679) Bad uneven segment distributions can happen after the rebalancing with SyncConsistentHashFactory
by Tristan Tarrant (Jira)
[ https://issues.redhat.com/browse/ISPN-11679?page=com.atlassian.jira.plugi... ]
Tristan Tarrant updated ISPN-11679:
-----------------------------------
Fix Version/s: 11.0.0.Final
> Bad uneven segment distributions can happen after the rebalancing with SyncConsistentHashFactory
> ------------------------------------------------------------------------------------------------
>
> Key: ISPN-11679
> URL: https://issues.redhat.com/browse/ISPN-11679
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.4.19.Final, 10.1.6.Final
> Reporter: Masafumi Miura
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 11.0.0.Final
>
>
> Uneven segment distribution can happen after rebalancing with SyncConsistentHashFactory.
> Even if a larger number of segments than "20 * number of nodes" is configured, some nodes have only around 50-60% number of segments than other nodes after the rebalance in the worst case.
> I understand the documentation (like the following) states about the possibility of slight uneven distribution as a potential drawback of SyncConsistentHashFactory.
> [Infinispan 10 configuration guide|https://infinispan.org/docs/stable/titles/configuring/configuring.h...] says:
> {quote}
> This implementation does have some negative points in that the load distribution is slightly uneven. It also moves more segments than strictly necessary on a join or leave.
> {quote}
> [JDG 7 Developer Guide|https://access.redhat.com/documentation/en-us/red_hat_data_grid/7.2...] says:
> {quote}
> It has a potential drawback in that it can move a greater number of segments than necessary during re-balancing. This can be mitigated by using a larger number of segments.
> Another potential drawback is that the segments are not distributed as evenly as possible, and actually using a very large number of segments can make the distribution of segments worse.
> {quote}
> However, only half segments than others is not a slight uneven.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months