[JBoss JIRA] (ISPN-11687) Infinispan not able to start after CR configuration change
by Vittorio Rigamonti (Jira)
[ https://issues.redhat.com/browse/ISPN-11687?page=com.atlassian.jira.plugi... ]
Vittorio Rigamonti updated ISPN-11687:
--------------------------------------
Status: Open (was: New)
> Infinispan not able to start after CR configuration change
> ----------------------------------------------------------
>
> Key: ISPN-11687
> URL: https://issues.redhat.com/browse/ISPN-11687
> Project: Infinispan
> Issue Type: Bug
> Components: OpenShift, Operator
> Affects Versions: 10.1.2.Final
> Reporter: Marian Macik
> Assignee: Vittorio Rigamonti
> Priority: Major
>
> Hi guys, as you may know, Kogito Operator uses Infinispan Operator to create a predefined Infinispan instance running on Openshift. Quite often we encounter an issue reproducible with these steps:
> 1. Install the Kogito Operator (this will also install Infinispan Operator).
> 2. Create a KogitoApp custom resource (CR) with this YAML:
> {code:yaml}
> apiVersion: app.kiegroup.org/v1alpha1
> kind: KogitoApp
> metadata:
> name: example-quarkus
> spec:
> enablePersistence: true
> build:
> envs:
> # enable persistence
> - name: MAVEN_ARGS_APPEND
> value: "-Ppersistence"
> gitSource:
> contextDir: process-quarkus-example
> uri: 'https://github.com/kiegroup/kogito-examples'
> reference: master
> {code}
> This will create a KogitoApp CR and will tell the Kogito Operator to provision Infinispan with one replica. Kogito application runs on Quarkus which makes use of RemoteCacheManager of Quarkus Infinispan Client Extension. Up to this point, everything works, application is deployed.
> 3. Try change Infinispan config by editing Infinispan CR a few times and Infinispan won't be able to start properly. By editing I mean - change one of these 3 parameters:
> {code:yaml}
> ...
> spec:
> container:
> cpu: ''
> extraJvmOpts: ''
> memory: ''
> ...
> {code}
> I generally want to change the cpu and memory as defaults are too low and I am also specifying `-Xmx2G` to extraJvmOpts so Infinispan has more heap than 200 MB which is default.
> Anyway, if you do this change a couple of times and after each change you wait until Infinispan pod is restarted, after ~5 times you will see [java.nio.channels.OverlappingFileLockException|https://gist.github.com/Ma...] in the Infinispan pod log.
> There is also another issue attached at the bottom of the Gist which was observed in Openshift events logs.
> What I have found is that if I create only KogitoInfra CR, which will create Infinispan CR and won't run any KogitoApp, so there is nothing connected to Infinispan, I can restart it how many times I want and it will work without any issues. I even tried to store something to the Infinispan via Infinispan REST API from the pod and tried changing Infinispan configuration then, and it worked like a charm after each restart.
> However, as soon as I deploy KogitoApp so it is connected to Infinispan using HotRod client and change Infinispan CR a few times (after each change waiting for Infinispan pod to restart), it will break with the linked exception present in the logs.
> To me it seems that this stops working once there is actual connection to Infinispan using HotRod client. I am not sure how this client works internally, but I would think that in addition to real user data there is some sort of exchange of "control data" let's say in this protocol between client and Infinispan which might break if Infinispan is suddenly restarted? Not sure, but with pushing there data using REST API (so without HotRod client) where the connection is maintained only for the time of the request, the exception didn't occur.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (ISPN-11556) SFS file location not relative to global persistent-location
by Will Burns (Jira)
[ https://issues.redhat.com/browse/ISPN-11556?page=com.atlassian.jira.plugi... ]
Will Burns resolved ISPN-11556.
-------------------------------
Resolution: Done
> SFS file location not relative to global persistent-location
> ------------------------------------------------------------
>
> Key: ISPN-11556
> URL: https://issues.redhat.com/browse/ISPN-11556
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration, Core
> Affects Versions: 10.1.5.Final, 11.0.0.Dev03
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Priority: Major
> Fix For: 10.1.7.Final, 11.0.0.Dev05
>
>
> Currently if a user specifies the location `/blah` in the SFS configuration, this location will be used regardless of the configure `persistent-location`. Instead, we should create the file location using `PersistenceUtil#getQualifiedLocation` and an exception will be thrown if `/blah` is not the root of the configure `persistent-location`.
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months
[JBoss JIRA] (ISPN-11695) Cannot create a Infinispan cluster with RHEL8.1
by Diego Lovison (Jira)
Diego Lovison created ISPN-11695:
------------------------------------
Summary: Cannot create a Infinispan cluster with RHEL8.1
Key: ISPN-11695
URL: https://issues.redhat.com/browse/ISPN-11695
Project: Infinispan
Issue Type: Bug
Affects Versions: 10.1.6.Final
Reporter: Diego Lovison
Open Terminal 1 and type: docker run infinispan/server
Open Terminal 2 and type: docker run infinispan/server
Cannot create an Infinispan cluster with RHEL8.1
*Terminal 1*
{noformat}
[jenkins@dlovison-rhel8-docker ~]$ docker run infinispan/server
17:11:56,272 INFO (main) [org.infinispan.SERVER] ISPN080000: Infinispan Server starting
17:11:56,273 INFO (main) [org.infinispan.SERVER] ISPN080017: Server configuration: /opt/infinispan/server/conf/infinispan.xml
17:11:56,273 INFO (main) [org.infinispan.SERVER] ISPN080032: Logging configuration: /opt/infinispan/server/conf/log4j2.xml
17:11:56,785 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'query-dsl-filter-converter-factory'
17:11:56,785 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'continuous-query-filter-converter-factory'
17:11:56,787 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'jdk.nashorn.api.scripting.NashornScriptEngineFactory'
17:11:57,538 INFO (main) [org.infinispan.CONTAINER] ISPN000128: Infinispan version: Infinispan 'Turia' 10.1.6.Final
17:11:57,716 INFO (main) [org.infinispan.CLUSTER] ISPN000078: Starting JGroups channel infinispan
17:12:02,840 INFO (main) [org.infinispan.CLUSTER] ISPN000094: Received new cluster view for channel infinispan: [85d6086e9784-1532|0] (1) [85d6086e9784-1532]
17:12:02,852 INFO (main) [org.infinispan.CLUSTER] ISPN000079: Channel infinispan local address is 85d6086e9784-1532, physical addresses are [172.17.0.2:7800]
17:12:02,876 INFO (main) [org.infinispan.CONTAINER] ISPN000390: Persisted state, version=10.1.6.Final timestamp=2020-04-22T17:12:02.874562Z
17:12:03,184 INFO (main) [org.infinispan.CONTAINER] ISPN000104: Using EmbeddedTransactionManager
17:12:03,628 INFO (ForkJoinPool.commonPool-worker-3) [org.infinispan.SERVER] ISPN080018: Protocol HotRod (internal)
17:12:03,734 INFO (main) [org.infinispan.SERVER] ISPN080018: Protocol REST (internal)
17:12:03,868 INFO (main) [org.infinispan.SERVER] ISPN080004: Protocol SINGLE_PORT listening on 172.17.0.2:11222
17:12:03,868 INFO (main) [org.infinispan.SERVER] ISPN080034: Server '85d6086e9784-1532' listening on http://172.17.0.2:11222
17:12:03,868 INFO (main) [org.infinispan.SERVER] ISPN080001: Infinispan Server 10.1.6.Final started in 7594ms
{noformat}
*Terminal 2*
{noformat}
[jenkins@dlovison-rhel8-docker ~]$ docker run infinispan/server
17:12:16,920 INFO (main) [org.infinispan.SERVER] ISPN080000: Infinispan Server starting
17:12:16,920 INFO (main) [org.infinispan.SERVER] ISPN080017: Server configuration: /opt/infinispan/server/conf/infinispan.xml
17:12:16,920 INFO (main) [org.infinispan.SERVER] ISPN080032: Logging configuration: /opt/infinispan/server/conf/log4j2.xml
17:12:17,425 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'query-dsl-filter-converter-factory'
17:12:17,425 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'continuous-query-filter-converter-factory'
17:12:17,426 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'jdk.nashorn.api.scripting.NashornScriptEngineFactory'
17:12:18,130 INFO (main) [org.infinispan.CONTAINER] ISPN000128: Infinispan version: Infinispan 'Turia' 10.1.6.Final
17:12:18,294 INFO (main) [org.infinispan.CLUSTER] ISPN000078: Starting JGroups channel infinispan
17:12:23,411 INFO (main) [org.infinispan.CLUSTER] ISPN000094: Received new cluster view for channel infinispan: [889155f7e9f5-21870|0] (1) [889155f7e9f5-21870]
17:12:23,424 INFO (main) [org.infinispan.CLUSTER] ISPN000079: Channel infinispan local address is 889155f7e9f5-21870, physical addresses are [172.17.0.3:7800]
17:12:23,458 INFO (main) [org.infinispan.CONTAINER] ISPN000390: Persisted state, version=10.1.6.Final timestamp=2020-04-22T17:12:23.455220Z
17:12:23,754 INFO (main) [org.infinispan.CONTAINER] ISPN000104: Using EmbeddedTransactionManager
17:12:24,147 INFO (ForkJoinPool.commonPool-worker-3) [org.infinispan.SERVER] ISPN080018: Protocol HotRod (internal)
17:12:24,218 INFO (main) [org.infinispan.SERVER] ISPN080018: Protocol REST (internal)
17:12:24,375 INFO (main) [org.infinispan.SERVER] ISPN080004: Protocol SINGLE_PORT listening on 172.17.0.3:11222
17:12:24,376 INFO (main) [org.infinispan.SERVER] ISPN080034: Server '889155f7e9f5-21870' listening on http://172.17.0.3:11222
17:12:24,376 INFO (main) [org.infinispan.SERVER] ISPN080001: Infinispan Server 10.1.6.Final started in 7455ms
{noformat}
It is working with RHEL7.7
*Terminal 1*
{noformat}
-bash-4.2$ hostname
dlovison-rhel7-docker
-bash-4.2$ docker run infinispan/server
17:19:29,766 INFO (main) [org.infinispan.SERVER] ISPN080000: Infinispan Server starting
17:19:29,766 INFO (main) [org.infinispan.SERVER] ISPN080017: Server configuration: /opt/infinispan/server/conf/infinispan.xml
17:19:29,767 INFO (main) [org.infinispan.SERVER] ISPN080032: Logging configuration: /opt/infinispan/server/conf/log4j2.xml
17:19:30,381 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'query-dsl-filter-converter-factory'
17:19:30,381 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'continuous-query-filter-converter-factory'
17:19:30,383 INFO (main) [org.infinispan.SERVER] ISPN080027: Loaded extension 'jdk.nashorn.api.scripting.NashornScriptEngineFactory'
17:19:31,343 INFO (main) [org.infinispan.CONTAINER] ISPN000128: Infinispan version: Infinispan 'Turia' 10.1.6.Final
17:19:31,590 INFO (main) [org.infinispan.CLUSTER] ISPN000078: Starting JGroups channel infinispan[Context=___hotRodTopologyCache_hotrod]ISPN100010: Finished rebalance with members [0d8d0f2b44a3-27743, 89cd86cd3964-63267], topology id 5
{noformat}
Terminal 2
{noformat}
-bash-4.2$ hostname
dlovison-rhel7-docker
-bash-4.2$ docker run infinispan/server
17:19:39,604 INFO (main) [org.infinispan.CLUSTER] ISPN000094: Received new cluster view for channel infinispan: [0d8d0f2b44a3-27743|1] (2) [0d8d0f2b44a3-27743, 89cd86cd3964-63267]
17:19:39,616 INFO (main) [org.infinispan.CLUSTER] ISPN000079: Channel infinispan local address is 89cd86cd3964-63267, physical addresses are [172.17.0.3:7800]
17:19:39,656 INFO (main) [org.infinispan.CONTAINER] ISPN000390: Persisted state, version=10.1.6.Final timestamp=2020-04-22T17:19:39.654715Z
17:19:39,979 INFO (main) [org.infinispan.CONTAINER] ISPN000104: Using EmbeddedTransactionManager
17:19:41,122 INFO (ForkJoinPool.commonPool-worker-3) [org.infinispan.SERVER] ISPN080018: Protocol HotRod (internal)
17:19:41,184 INFO (main) [org.infinispan.SERVER] ISPN080018: Protocol REST (internal)
17:19:41,356 INFO (main) [org.infinispan.SERVER] ISPN080004: Protocol SINGLE_PORT listening on 172.17.0.3:11222
17:19:41,356 INFO (main) [org.infinispan.SERVER] ISPN080034: Server '89cd86cd3964-63267' listening on http://172.17.0.3:11222
17:19:41,356 INFO (main) [org.infinispan.SERVER] ISPN080001: Infinispan Server 10.1.6.Final started in 8620ms
{noformat}
--
This message was sent by Atlassian Jira
(v7.13.8#713008)
5 years, 11 months