I think you can't mix old and new Keycloak servers in same cluster. And rolling upgrade (zero downtime upgrade) is not yet supported. We plan to add support for it, but it won't be in a very near future as it will likely require quite a lot of work... In shortcut, it will be recommended to stop all pods with 4.5.0 and then start pods with 6.0.1. Marek On 02/07/2019 16:25, Мартынов Илья wrote: > Hello! > I have Keycloak 4.5.0.Final deployed in standalone-ha configuration in k8s > cluster. When I try to update Keycloak to version 6.0.1, the following > happens: > 1. K8s starts new pod with version 6.0.1 > 2. Old pod still running, it will be terminated on successfull readiness > probe of the new pod > 3. New pod discovers old pod via JGroups, cache synchronization started > 4. Exception in new pod: > 02-07-2019;13:34:29,220 WARN [stateTransferExecutor-thread--p25-t1] > org.infinispan.statetransfer.InboundTransferTask ISPN000210: Failed to > request state of cache work from node idp-6569c544b > -hsd6g, segments {0-255}: org.infinispan.remoting.RemoteException: > ISPN000217: Received exception from idp-6569c544b-hsd6g, see cause for > remote stack trace > at org.infinispan(a)9.4.8.Final > //org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:28) > ... > Caused by: java.io.IOException: Unknown type: 132 > at > org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:681) > at > org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:355) > at > org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40) > > Looks like this exception blocks further Keycloak startup, because nothing > happens in logs afterwards. Also, my rest service deployed as JAX-RS bean > also doesn't respond, so pod is not treated as alive by Kubernetes. > Please help. > _______________________________________________ > keycloak-dev mailing list > keycloak-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/keycloak-dev

Just another question. Actually, I don't even need zero-downtime upgrade, I need just any upgrade. Now upgrade is blocked because new pod cannot start. Can we somehow make new pod to ignore non-readable messages from old pod and continue loading? Yes, session state won't be replicated, but it's more automated way then manual stop of old pod and starting a new pod after that. ср, 3 июл. 2019 г. в 09:39, Мартынов Илья <imartynovsp(a)gmail.com&gt;: > Hi Marek, > > Ok, I got it, thank you for response! > > вт, 2 июля 2019 г., 19:25 Marek Posolda <mposolda(a)redhat.com&gt;: > >> I think you can't mix old and new Keycloak servers in same cluster. And >> rolling upgrade (zero downtime upgrade) is not yet supported. We plan to >> add support for it, but it won't be in a very near future as it will >> likely require quite a lot of work... >> >> In shortcut, it will be recommended to stop all pods with 4.5.0 and then >> start pods with 6.0.1. >> >> Marek >> >> >> On 02/07/2019 16:25, Мартынов Илья wrote: >> > Hello! >> > I have Keycloak 4.5.0.Final deployed in standalone-ha configuration in >> k8s >> > cluster. When I try to update Keycloak to version 6.0.1, the following >> > happens: >> > 1. K8s starts new pod with version 6.0.1 >> > 2. Old pod still running, it will be terminated on successfull readiness >> > probe of the new pod >> > 3. New pod discovers old pod via JGroups, cache synchronization started >> > 4. Exception in new pod: >> > 02-07-2019;13:34:29,220 WARN [stateTransferExecutor-thread--p25-t1] >> > org.infinispan.statetransfer.InboundTransferTask ISPN000210: Failed to >> > request state of cache work from node idp-6569c544b >> > -hsd6g, segments {0-255}: org.infinispan.remoting.RemoteException: >> > ISPN000217: Received exception from idp-6569c544b-hsd6g, see cause for >> > remote stack trace >> > at org.infinispan(a)9.4.8.Final >> > >> //org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:28) >> > ... >> > Caused by: java.io.IOException: Unknown type: 132 >> > at >> > >> org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:681) >> > at >> > >> org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:355) >> > at >> > >> org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40) >> > >> > Looks like this exception blocks further Keycloak startup, because >> nothing >> > happens in logs afterwards. Also, my rest service deployed as JAX-RS >> bean >> > also doesn't respond, so pod is not treated as alive by Kubernetes. >> > Please help. >> > _______________________________________________ >> > keycloak-dev mailing list >> > keycloak-dev(a)lists.jboss.org >> > https://lists.jboss.org/mailman/listinfo/keycloak-dev >> >> >> _______________________________________________ keycloak-dev mailing list keycloak-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/keycloak-dev

Thanks Sebastian, I use Kube_ping JGroups discovery plugin, it seems to be more reliable than IP Multicasting. Finally I've utilized choise #1 simply changing Strategy of Deployment in kubernetes from default "RollingUpdate" to "Recreate". Recreate does exactly what we need, first drops all existing pods and after that creates pods with new version. ср, 3 июл. 2019 г. в 15:15, Sebastian Laskawiec <slaskawi(a)redhat.com&gt;: > If you're using standalone-ha.xml without any extra parameters, you're > using UDP JGroups stack with IP Multicasting discovery. You also suggested > that you're using Pods, so I'm assuming you're using Kubernetes with > Flannel network plugin (as far as I know, other plugins do not support IP > Multicasting out of the box). > > So effectively you have 2 choices: > - Turn everything off, do the migration and start it again. Note, that > you need to shut everything down (this is not a rolling update procedure). > - Reconfigure JGroups not to join the same cluster. The easiest thing to > do here is to modify the cluster attribute and make sure it's different for > the old and new cluster. > > Thanks, > Sebastian > > On Wed, Jul 3, 2019 at 11:10 AM Мартынов Илья <imartynovsp(a)gmail.com&gt; > wrote: > >> Just another question. Actually, I don't even need zero-downtime >> upgrade, I >> need just any upgrade. Now upgrade is blocked because new pod cannot >> start. >> Can we somehow make new pod to ignore non-readable messages from old pod >> and continue loading? >> Yes, session state won't be replicated, but it's more automated way then >> manual stop of old pod and starting a new pod after that. >> >> >> >> ср, 3 июл. 2019 г. в 09:39, Мартынов Илья <imartynovsp(a)gmail.com&gt;: >> >> > Hi Marek, >> > >> > Ok, I got it, thank you for response! >> > >> > вт, 2 июля 2019 г., 19:25 Marek Posolda <mposolda(a)redhat.com&gt;: >> > >> >> I think you can't mix old and new Keycloak servers in same cluster. >> And >> >> rolling upgrade (zero downtime upgrade) is not yet supported. We plan >> to >> >> add support for it, but it won't be in a very near future as it will >> >> likely require quite a lot of work... >> >> >> >> In shortcut, it will be recommended to stop all pods with 4.5.0 and >> then >> >> start pods with 6.0.1. >> >> >> >> Marek >> >> >> >> >> >> On 02/07/2019 16:25, Мартынов Илья wrote: >> >> > Hello! >> >> > I have Keycloak 4.5.0.Final deployed in standalone-ha configuration >> in >> >> k8s >> >> > cluster. When I try to update Keycloak to version 6.0.1, the >> following >> >> > happens: >> >> > 1. K8s starts new pod with version 6.0.1 >> >> > 2. Old pod still running, it will be terminated on successfull >> readiness >> >> > probe of the new pod >> >> > 3. New pod discovers old pod via JGroups, cache synchronization >> started >> >> > 4. Exception in new pod: >> >> > 02-07-2019;13:34:29,220 WARN [stateTransferExecutor-thread--p25-t1] >> >> > org.infinispan.statetransfer.InboundTransferTask ISPN000210: Failed >> to >> >> > request state of cache work from node idp-6569c544b >> >> > -hsd6g, segments {0-255}: org.infinispan.remoting.RemoteException: >> >> > ISPN000217: Received exception from idp-6569c544b-hsd6g, see cause >> for >> >> > remote stack trace >> >> > at org.infinispan(a)9.4.8.Final >> >> > >> >> >> //org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:28) >> >> > ... >> >> > Caused by: java.io.IOException: Unknown type: 132 >> >> > at >> >> > >> >> >> org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:681) >> >> > at >> >> > >> >> >> org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:355) >> >> > at >> >> > >> >> >> org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40) >> >> > >> >> > Looks like this exception blocks further Keycloak startup, because >> >> nothing >> >> > happens in logs afterwards. Also, my rest service deployed as JAX-RS >> >> bean >> >> > also doesn't respond, so pod is not treated as alive by Kubernetes. >> >> > Please help. >> >> > _______________________________________________ >> >> > keycloak-dev mailing list >> >> > keycloak-dev(a)lists.jboss.org >> >> > https://lists.jboss.org/mailman/listinfo/keycloak-dev >> >> >> >> >> >> >> _______________________________________________ >> keycloak-dev mailing list >> keycloak-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/keycloak-dev > >

I've tried split_clusters_during_rolling_update option, but KUBE_PING start failing with NPE: Caused by: java.lang.NullPointerException at java.util.Objects.requireNonNull(Objects.java:203) at java.util.Optional.<init>(Optional.java:96) at java.util.Optional.of(Optional.java:108) at java.util.stream.FindOps$FindSink$OfRef.get(FindOps.java:193) at java.util.stream.FindOps$FindSink$OfRef.get(FindOps.java:190) at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464) at org.jgroups.protocols.kubernetes.KUBE_PING.findMembers(KUBE_PING.java:240) at org.jgroups.protocols.Discovery.findMembers(Discovery.java:211) at org.jgroups.protocols.Discovery.down(Discovery.java:350) In debug I see code in KUBE_PING that tries to find pod's deployment String senderParentDeployment = hosts.stream() .filter(pod -> senderIp.contains(pod.getIp())) .map(Pod::getParentDeployment) .findFirst().orElse(null); This code fails with NPE, as Pod::getParentDeployment returns null. Parent deployment not set because KUBE_PING cannot read deployment from json describing pod. There is already a bug on KUBE_PING for it: https://github.com/jgroups-extras/jgroups-kubernetes/issues/50 пн, 8 июл. 2019 г. в 10:25, Sebastian Laskawiec <slaskawi(a)redhat.com&gt;: > Ok, I'm glad it worked. > > Just for the future - with KUBE_PING, there's an additional option: > "split_clusters_during_rolling_update" set to true. See the documentation: > https://github.com/jgroups-extras/jgroups-kubernetes#kube_ping-configuration > > On Wed, Jul 3, 2019 at 4:32 PM Мартынов Илья <imartynovsp(a)gmail.com&gt; > wrote: > >> Thanks Sebastian, >> I use Kube_ping JGroups discovery plugin, it seems to be more reliable >> than IP Multicasting. >> Finally I've utilized choise #1 simply changing Strategy of Deployment >> in kubernetes from default "RollingUpdate" to "Recreate". Recreate does >> exactly what we need, first drops all existing pods and after that creates >> pods with new version. >> >> >> >> ср, 3 июл. 2019 г. в 15:15, Sebastian Laskawiec <slaskawi(a)redhat.com&gt;: >> >>> If you're using standalone-ha.xml without any extra parameters, you're >>> using UDP JGroups stack with IP Multicasting discovery. You also suggested >>> that you're using Pods, so I'm assuming you're using Kubernetes with >>> Flannel network plugin (as far as I know, other plugins do not support IP >>> Multicasting out of the box). >>> >>> So effectively you have 2 choices: >>> - Turn everything off, do the migration and start it again. Note, that >>> you need to shut everything down (this is not a rolling update procedure). >>> - Reconfigure JGroups not to join the same cluster. The easiest thing >>> to do here is to modify the cluster attribute and make sure it's different >>> for the old and new cluster. >>> >>> Thanks, >>> Sebastian >>> >>> On Wed, Jul 3, 2019 at 11:10 AM Мартынов Илья <imartynovsp(a)gmail.com&gt; >>> wrote: >>> >>>> Just another question. Actually, I don't even need zero-downtime >>>> upgrade, I >>>> need just any upgrade. Now upgrade is blocked because new pod cannot >>>> start. >>>> Can we somehow make new pod to ignore non-readable messages from old >>>> pod >>>> and continue loading? >>>> Yes, session state won't be replicated, but it's more automated way >>>> then >>>> manual stop of old pod and starting a new pod after that. >>>> >>>> >>>> >>>> ср, 3 июл. 2019 г. в 09:39, Мартынов Илья <imartynovsp(a)gmail.com&gt;: >>>> >>>> > Hi Marek, >>>> > >>>> > Ok, I got it, thank you for response! >>>> > >>>> > вт, 2 июля 2019 г., 19:25 Marek Posolda <mposolda(a)redhat.com&gt;: >>>> > >>>> >> I think you can't mix old and new Keycloak servers in same cluster. >>>> And >>>> >> rolling upgrade (zero downtime upgrade) is not yet supported. We >>>> plan to >>>> >> add support for it, but it won't be in a very near future as it will >>>> >> likely require quite a lot of work... >>>> >> >>>> >> In shortcut, it will be recommended to stop all pods with 4.5.0 and >>>> then >>>> >> start pods with 6.0.1. >>>> >> >>>> >> Marek >>>> >> >>>> >> >>>> >> On 02/07/2019 16:25, Мартынов Илья wrote: >>>> >> > Hello! >>>> >> > I have Keycloak 4.5.0.Final deployed in standalone-ha >>>> configuration in >>>> >> k8s >>>> >> > cluster. When I try to update Keycloak to version 6.0.1, the >>>> following >>>> >> > happens: >>>> >> > 1. K8s starts new pod with version 6.0.1 >>>> >> > 2. Old pod still running, it will be terminated on successfull >>>> readiness >>>> >> > probe of the new pod >>>> >> > 3. New pod discovers old pod via JGroups, cache synchronization >>>> started >>>> >> > 4. Exception in new pod: >>>> >> > 02-07-2019;13:34:29,220 WARN >>>> [stateTransferExecutor-thread--p25-t1] >>>> >> > org.infinispan.statetransfer.InboundTransferTask ISPN000210: >>>> Failed to >>>> >> > request state of cache work from node idp-6569c544b >>>> >> > -hsd6g, segments {0-255}: org.infinispan.remoting.RemoteException: >>>> >> > ISPN000217: Received exception from idp-6569c544b-hsd6g, see >>>> cause for >>>> >> > remote stack trace >>>> >> > at org.infinispan(a)9.4.8.Final >>>> >> > >>>> >> >>>> //org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:28) >>>> >> > ... >>>> >> > Caused by: java.io.IOException: Unknown type: 132 >>>> >> > at >>>> >> > >>>> >> >>>> org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:681) >>>> >> > at >>>> >> > >>>> >> >>>> org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:355) >>>> >> > at >>>> >> > >>>> >> >>>> org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40) >>>> >> > >>>> >> > Looks like this exception blocks further Keycloak startup, because >>>> >> nothing >>>> >> > happens in logs afterwards. Also, my rest service deployed as >>>> JAX-RS >>>> >> bean >>>> >> > also doesn't respond, so pod is not treated as alive by >>>> Kubernetes. >>>> >> > Please help. >>>> >> > _______________________________________________ >>>> >> > keycloak-dev mailing list >>>> >> > keycloak-dev(a)lists.jboss.org >>>> >> > https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>> >> >>>> >> >>>> >> >>>> _______________________________________________ >>>> keycloak-dev mailing list >>>> keycloak-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>> >>>

I've estimated the support of ReplicaSets and it looks like possible to implement in 1 day. So yes, I start working on it. пн, 8 июл. 2019 г. в 15:28, Sebastian Laskawiec <slaskawi(a)redhat.com&gt;: > oooh right, I guess you hit this issue: > https://github.com/jgroups-extras/jgroups-kubernetes/issues/50 > > Sorry to hear that. Perhaps you'd be interested in contributing a fix to > KUBE_PING? > > On Mon, Jul 8, 2019 at 1:35 PM Мартынов Илья <imartynovsp(a)gmail.com&gt; > wrote: > >> I've tried split_clusters_during_rolling_update option, but KUBE_PING >> start failing with NPE: >> Caused by: java.lang.NullPointerException >> at java.util.Objects.requireNonNull(Objects.java:203) >> at java.util.Optional.<init>(Optional.java:96) >> at java.util.Optional.of(Optional.java:108) >> at java.util.stream.FindOps$FindSink$OfRef.get(FindOps.java:193) >> at java.util.stream.FindOps$FindSink$OfRef.get(FindOps.java:190) >> at >> java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152) >> at >> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) >> at >> java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464) >> at >> org.jgroups.protocols.kubernetes.KUBE_PING.findMembers(KUBE_PING.java:240) >> at >> org.jgroups.protocols.Discovery.findMembers(Discovery.java:211) >> at org.jgroups.protocols.Discovery.down(Discovery.java:350) >> >> In debug I see code in KUBE_PING that tries to find pod's deployment >> String senderParentDeployment = hosts.stream() >> .filter(pod -> senderIp.contains(pod.getIp())) >> .map(Pod::getParentDeployment) >> .findFirst().orElse(null); >> This code fails with NPE, as Pod::getParentDeployment returns null. >> Parent deployment not set because KUBE_PING cannot read deployment from >> json describing pod. >> There is already a bug on KUBE_PING for it: >> https://github.com/jgroups-extras/jgroups-kubernetes/issues/50 >> >> >> пн, 8 июл. 2019 г. в 10:25, Sebastian Laskawiec <slaskawi(a)redhat.com&gt;: >> >>> Ok, I'm glad it worked. >>> >>> Just for the future - with KUBE_PING, there's an additional option: >>> "split_clusters_during_rolling_update" set to true. See the documentation: >>> https://github.com/jgroups-extras/jgroups-kubernetes#kube_ping-configuration >>> >>> On Wed, Jul 3, 2019 at 4:32 PM Мартынов Илья <imartynovsp(a)gmail.com&gt; >>> wrote: >>> >>>> Thanks Sebastian, >>>> I use Kube_ping JGroups discovery plugin, it seems to be more reliable >>>> than IP Multicasting. >>>> Finally I've utilized choise #1 simply changing Strategy of Deployment >>>> in kubernetes from default "RollingUpdate" to "Recreate". Recreate does >>>> exactly what we need, first drops all existing pods and after that creates >>>> pods with new version. >>>> >>>> >>>> >>>> ср, 3 июл. 2019 г. в 15:15, Sebastian Laskawiec <slaskawi(a)redhat.com&gt;: >>>> >>>>> If you're using standalone-ha.xml without any extra parameters, >>>>> you're using UDP JGroups stack with IP Multicasting discovery. You also >>>>> suggested that you're using Pods, so I'm assuming you're using Kubernetes >>>>> with Flannel network plugin (as far as I know, other plugins do not support >>>>> IP Multicasting out of the box). >>>>> >>>>> So effectively you have 2 choices: >>>>> - Turn everything off, do the migration and start it again. Note, >>>>> that you need to shut everything down (this is not a rolling update >>>>> procedure). >>>>> - Reconfigure JGroups not to join the same cluster. The easiest thing >>>>> to do here is to modify the cluster attribute and make sure it's different >>>>> for the old and new cluster. >>>>> >>>>> Thanks, >>>>> Sebastian >>>>> >>>>> On Wed, Jul 3, 2019 at 11:10 AM Мартынов Илья <imartynovsp(a)gmail.com&gt; >>>>> wrote: >>>>> >>>>>> Just another question. Actually, I don't even need zero-downtime >>>>>> upgrade, I >>>>>> need just any upgrade. Now upgrade is blocked because new pod cannot >>>>>> start. >>>>>> Can we somehow make new pod to ignore non-readable messages from old >>>>>> pod >>>>>> and continue loading? >>>>>> Yes, session state won't be replicated, but it's more automated way >>>>>> then >>>>>> manual stop of old pod and starting a new pod after that. >>>>>> >>>>>> >>>>>> >>>>>> ср, 3 июл. 2019 г. в 09:39, Мартынов Илья <imartynovsp(a)gmail.com&gt;: >>>>>> >>>>>> > Hi Marek, >>>>>> > >>>>>> > Ok, I got it, thank you for response! >>>>>> > >>>>>> > вт, 2 июля 2019 г., 19:25 Marek Posolda <mposolda(a)redhat.com&gt;: >>>>>> > >>>>>> >> I think you can't mix old and new Keycloak servers in same >>>>>> cluster. And >>>>>> >> rolling upgrade (zero downtime upgrade) is not yet supported. We >>>>>> plan to >>>>>> >> add support for it, but it won't be in a very near future as it >>>>>> will >>>>>> >> likely require quite a lot of work... >>>>>> >> >>>>>> >> In shortcut, it will be recommended to stop all pods with 4.5.0 >>>>>> and then >>>>>> >> start pods with 6.0.1. >>>>>> >> >>>>>> >> Marek >>>>>> >> >>>>>> >> >>>>>> >> On 02/07/2019 16:25, Мартынов Илья wrote: >>>>>> >> > Hello! >>>>>> >> > I have Keycloak 4.5.0.Final deployed in standalone-ha >>>>>> configuration in >>>>>> >> k8s >>>>>> >> > cluster. When I try to update Keycloak to version 6.0.1, the >>>>>> following >>>>>> >> > happens: >>>>>> >> > 1. K8s starts new pod with version 6.0.1 >>>>>> >> > 2. Old pod still running, it will be terminated on successfull >>>>>> readiness >>>>>> >> > probe of the new pod >>>>>> >> > 3. New pod discovers old pod via JGroups, cache synchronization >>>>>> started >>>>>> >> > 4. Exception in new pod: >>>>>> >> > 02-07-2019;13:34:29,220 WARN >>>>>> [stateTransferExecutor-thread--p25-t1] >>>>>> >> > org.infinispan.statetransfer.InboundTransferTask ISPN000210: >>>>>> Failed to >>>>>> >> > request state of cache work from node idp-6569c544b >>>>>> >> > -hsd6g, segments {0-255}: >>>>>> org.infinispan.remoting.RemoteException: >>>>>> >> > ISPN000217: Received exception from idp-6569c544b-hsd6g, see >>>>>> cause for >>>>>> >> > remote stack trace >>>>>> >> > at org.infinispan(a)9.4.8.Final >>>>>> >> > >>>>>> >> >>>>>> //org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:28) >>>>>> >> > ... >>>>>> >> > Caused by: java.io.IOException: Unknown type: 132 >>>>>> >> > at >>>>>> >> > >>>>>> >> >>>>>> org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:681) >>>>>> >> > at >>>>>> >> > >>>>>> >> >>>>>> org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:355) >>>>>> >> > at >>>>>> >> > >>>>>> >> >>>>>> org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40) >>>>>> >> > >>>>>> >> > Looks like this exception blocks further Keycloak startup, >>>>>> because >>>>>> >> nothing >>>>>> >> > happens in logs afterwards. Also, my rest service deployed as >>>>>> JAX-RS >>>>>> >> bean >>>>>> >> > also doesn't respond, so pod is not treated as alive by >>>>>> Kubernetes. >>>>>> >> > Please help. >>>>>> >> > _______________________________________________ >>>>>> >> > keycloak-dev mailing list >>>>>> >> > keycloak-dev(a)lists.jboss.org >>>>>> >> > https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> _______________________________________________ >>>>>> keycloak-dev mailing list >>>>>> keycloak-dev(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>> >>>>>

Thanks a lot for the patch. Let's move the conversation to the Pull Request. On Mon, Jul 8, 2019 at 5:47 PM Мартынов Илья <imartynovsp(a)gmail.com&gt; wrote: > Created https://github.com/jgroups-extras/jgroups-kubernetes/pull/72 - > please take a look! > Not clear for me who and how will modify KUBE_PING jboss module, it > requires new dependecy now for JsonPath processing. > > пн, 8 июл. 2019 г. в 17:31, Мартынов Илья <imartynovsp(a)gmail.com&gt;: > >> I've estimated the support of ReplicaSets and it looks like possible to >> implement in 1 day. So yes, I start working on it. >> >> пн, 8 июл. 2019 г. в 15:28, Sebastian Laskawiec <slaskawi(a)redhat.com&gt;: >> >>> oooh right, I guess you hit this issue: >>> https://github.com/jgroups-extras/jgroups-kubernetes/issues/50 >>> >>> Sorry to hear that. Perhaps you'd be interested in contributing a fix >>> to KUBE_PING? >>> >>> On Mon, Jul 8, 2019 at 1:35 PM Мартынов Илья <imartynovsp(a)gmail.com&gt; >>> wrote: >>> >>>> I've tried split_clusters_during_rolling_update option, but KUBE_PING >>>> start failing with NPE: >>>> Caused by: java.lang.NullPointerException >>>> at java.util.Objects.requireNonNull(Objects.java:203) >>>> at java.util.Optional.<init>(Optional.java:96) >>>> at java.util.Optional.of(Optional.java:108) >>>> at >>>> java.util.stream.FindOps$FindSink$OfRef.get(FindOps.java:193) >>>> at >>>> java.util.stream.FindOps$FindSink$OfRef.get(FindOps.java:190) >>>> at >>>> java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152) >>>> at >>>> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) >>>> at >>>> java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464) >>>> at >>>> org.jgroups.protocols.kubernetes.KUBE_PING.findMembers(KUBE_PING.java:240) >>>> at >>>> org.jgroups.protocols.Discovery.findMembers(Discovery.java:211) >>>> at org.jgroups.protocols.Discovery.down(Discovery.java:350) >>>> >>>> In debug I see code in KUBE_PING that tries to find pod's deployment >>>> String senderParentDeployment = hosts.stream() >>>> .filter(pod -> senderIp.contains(pod.getIp())) >>>> .map(Pod::getParentDeployment) >>>> .findFirst().orElse(null); >>>> This code fails with NPE, as Pod::getParentDeployment returns null. >>>> Parent deployment not set because KUBE_PING cannot read deployment from >>>> json describing pod. >>>> There is already a bug on KUBE_PING for it: >>>> https://github.com/jgroups-extras/jgroups-kubernetes/issues/50 >>>> >>>> >>>> пн, 8 июл. 2019 г. в 10:25, Sebastian Laskawiec <slaskawi(a)redhat.com&gt;: >>>> >>>>> Ok, I'm glad it worked. >>>>> >>>>> Just for the future - with KUBE_PING, there's an additional option: >>>>> "split_clusters_during_rolling_update" set to true. See the documentation: >>>>> https://github.com/jgroups-extras/jgroups-kubernetes#kube_ping-configuration >>>>> >>>>> On Wed, Jul 3, 2019 at 4:32 PM Мартынов Илья <imartynovsp(a)gmail.com&gt; >>>>> wrote: >>>>> >>>>>> Thanks Sebastian, >>>>>> I use Kube_ping JGroups discovery plugin, it seems to be more >>>>>> reliable than IP Multicasting. >>>>>> Finally I've utilized choise #1 simply changing Strategy of >>>>>> Deployment in kubernetes from default "RollingUpdate" to "Recreate". >>>>>> Recreate does exactly what we need, first drops all existing pods and after >>>>>> that creates pods with new version. >>>>>> >>>>>> >>>>>> >>>>>> ср, 3 июл. 2019 г. в 15:15, Sebastian Laskawiec <slaskawi(a)redhat.com >>>>>> >: >>>>>> >>>>>>> If you're using standalone-ha.xml without any extra parameters, >>>>>>> you're using UDP JGroups stack with IP Multicasting discovery. You also >>>>>>> suggested that you're using Pods, so I'm assuming you're using Kubernetes >>>>>>> with Flannel network plugin (as far as I know, other plugins do not support >>>>>>> IP Multicasting out of the box). >>>>>>> >>>>>>> So effectively you have 2 choices: >>>>>>> - Turn everything off, do the migration and start it again. Note, >>>>>>> that you need to shut everything down (this is not a rolling update >>>>>>> procedure). >>>>>>> - Reconfigure JGroups not to join the same cluster. The easiest >>>>>>> thing to do here is to modify the cluster attribute and make sure it's >>>>>>> different for the old and new cluster. >>>>>>> >>>>>>> Thanks, >>>>>>> Sebastian >>>>>>> >>>>>>> On Wed, Jul 3, 2019 at 11:10 AM Мартынов Илья < >>>>>>> imartynovsp(a)gmail.com&gt; wrote: >>>>>>> >>>>>>>> Just another question. Actually, I don't even need zero-downtime >>>>>>>> upgrade, I >>>>>>>> need just any upgrade. Now upgrade is blocked because new pod >>>>>>>> cannot start. >>>>>>>> Can we somehow make new pod to ignore non-readable messages from >>>>>>>> old pod >>>>>>>> and continue loading? >>>>>>>> Yes, session state won't be replicated, but it's more automated >>>>>>>> way then >>>>>>>> manual stop of old pod and starting a new pod after that. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ср, 3 июл. 2019 г. в 09:39, Мартынов Илья <imartynovsp(a)gmail.com&gt;: >>>>>>>> >>>>>>>> > Hi Marek, >>>>>>>> > >>>>>>>> > Ok, I got it, thank you for response! >>>>>>>> > >>>>>>>> > вт, 2 июля 2019 г., 19:25 Marek Posolda <mposolda(a)redhat.com&gt;: >>>>>>>> > >>>>>>>> >> I think you can't mix old and new Keycloak servers in same >>>>>>>> cluster. And >>>>>>>> >> rolling upgrade (zero downtime upgrade) is not yet supported. >>>>>>>> We plan to >>>>>>>> >> add support for it, but it won't be in a very near future as it >>>>>>>> will >>>>>>>> >> likely require quite a lot of work... >>>>>>>> >> >>>>>>>> >> In shortcut, it will be recommended to stop all pods with 4.5.0 >>>>>>>> and then >>>>>>>> >> start pods with 6.0.1. >>>>>>>> >> >>>>>>>> >> Marek >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> On 02/07/2019 16:25, Мартынов Илья wrote: >>>>>>>> >> > Hello! >>>>>>>> >> > I have Keycloak 4.5.0.Final deployed in standalone-ha >>>>>>>> configuration in >>>>>>>> >> k8s >>>>>>>> >> > cluster. When I try to update Keycloak to version 6.0.1, the >>>>>>>> following >>>>>>>> >> > happens: >>>>>>>> >> > 1. K8s starts new pod with version 6.0.1 >>>>>>>> >> > 2. Old pod still running, it will be terminated on >>>>>>>> successfull readiness >>>>>>>> >> > probe of the new pod >>>>>>>> >> > 3. New pod discovers old pod via JGroups, cache >>>>>>>> synchronization started >>>>>>>> >> > 4. Exception in new pod: >>>>>>>> >> > 02-07-2019;13:34:29,220 WARN >>>>>>>> [stateTransferExecutor-thread--p25-t1] >>>>>>>> >> > org.infinispan.statetransfer.InboundTransferTask ISPN000210: >>>>>>>> Failed to >>>>>>>> >> > request state of cache work from node idp-6569c544b >>>>>>>> >> > -hsd6g, segments {0-255}: >>>>>>>> org.infinispan.remoting.RemoteException: >>>>>>>> >> > ISPN000217: Received exception from idp-6569c544b-hsd6g, see >>>>>>>> cause for >>>>>>>> >> > remote stack trace >>>>>>>> >> > at org.infinispan(a)9.4.8.Final >>>>>>>> >> > >>>>>>>> >> >>>>>>>> //org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:28) >>>>>>>> >> > ... >>>>>>>> >> > Caused by: java.io.IOException: Unknown type: 132 >>>>>>>> >> > at >>>>>>>> >> > >>>>>>>> >> >>>>>>>> org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:681) >>>>>>>> >> > at >>>>>>>> >> > >>>>>>>> >> >>>>>>>> org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:355) >>>>>>>> >> > at >>>>>>>> >> > >>>>>>>> >> >>>>>>>> org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40) >>>>>>>> >> > >>>>>>>> >> > Looks like this exception blocks further Keycloak startup, >>>>>>>> because >>>>>>>> >> nothing >>>>>>>> >> > happens in logs afterwards. Also, my rest service deployed as >>>>>>>> JAX-RS >>>>>>>> >> bean >>>>>>>> >> > also doesn't respond, so pod is not treated as alive by >>>>>>>> Kubernetes. >>>>>>>> >> > Please help. >>>>>>>> >> > _______________________________________________ >>>>>>>> >> > keycloak-dev mailing list >>>>>>>> >> > keycloak-dev(a)lists.jboss.org >>>>>>>> >> > https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> _______________________________________________ >>>>>>>> keycloak-dev mailing list >>>>>>>> keycloak-dev(a)lists.jboss.org >>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev >>>>>>> >>>>>>>