[keycloak-dev] Infinispan error during update in ha configuration
Sebastian Laskawiec
slaskawi at redhat.com
Thu Aug 1 04:07:45 EDT 2019
Thanks a lot for the contribution of this feature. It is now available in
1.0.12.Final release:
https://github.com/jgroups-extras/jgroups-kubernetes/releases/tag/1.0.12.Final
The bits should appear in Maven Central as soon as the sync finishes. In
the meantime, you might grab them here:
https://repository.jboss.org/nexus/content/repositories/releases/org/jgroups/kubernetes/jgroups-kubernetes/1.0.12.Final/
On Tue, Jul 9, 2019 at 9:55 AM Sebastian Laskawiec <slaskawi at redhat.com>
wrote:
> Thanks a lot for the patch. Let's move the conversation to the Pull
> Request.
>
> On Mon, Jul 8, 2019 at 5:47 PM Мартынов Илья <imartynovsp at gmail.com>
> wrote:
>
>> Created https://github.com/jgroups-extras/jgroups-kubernetes/pull/72 -
>> please take a look!
>> Not clear for me who and how will modify KUBE_PING jboss module, it
>> requires new dependecy now for JsonPath processing.
>>
>> пн, 8 июл. 2019 г. в 17:31, Мартынов Илья <imartynovsp at gmail.com>:
>>
>>> I've estimated the support of ReplicaSets and it looks like possible to
>>> implement in 1 day. So yes, I start working on it.
>>>
>>> пн, 8 июл. 2019 г. в 15:28, Sebastian Laskawiec <slaskawi at redhat.com>:
>>>
>>>> oooh right, I guess you hit this issue:
>>>> https://github.com/jgroups-extras/jgroups-kubernetes/issues/50
>>>>
>>>> Sorry to hear that. Perhaps you'd be interested in contributing a fix
>>>> to KUBE_PING?
>>>>
>>>> On Mon, Jul 8, 2019 at 1:35 PM Мартынов Илья <imartynovsp at gmail.com>
>>>> wrote:
>>>>
>>>>> I've tried split_clusters_during_rolling_update option, but KUBE_PING
>>>>> start failing with NPE:
>>>>> Caused by: java.lang.NullPointerException
>>>>> at java.util.Objects.requireNonNull(Objects.java:203)
>>>>> at java.util.Optional.<init>(Optional.java:96)
>>>>> at java.util.Optional.of(Optional.java:108)
>>>>> at
>>>>> java.util.stream.FindOps$FindSink$OfRef.get(FindOps.java:193)
>>>>> at
>>>>> java.util.stream.FindOps$FindSink$OfRef.get(FindOps.java:190)
>>>>> at
>>>>> java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
>>>>> at
>>>>> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
>>>>> at
>>>>> java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464)
>>>>> at
>>>>> org.jgroups.protocols.kubernetes.KUBE_PING.findMembers(KUBE_PING.java:240)
>>>>> at
>>>>> org.jgroups.protocols.Discovery.findMembers(Discovery.java:211)
>>>>> at org.jgroups.protocols.Discovery.down(Discovery.java:350)
>>>>>
>>>>> In debug I see code in KUBE_PING that tries to find pod's deployment
>>>>> String senderParentDeployment = hosts.stream()
>>>>> .filter(pod -> senderIp.contains(pod.getIp()))
>>>>> .map(Pod::getParentDeployment)
>>>>> .findFirst().orElse(null);
>>>>> This code fails with NPE, as Pod::getParentDeployment returns null.
>>>>> Parent deployment not set because KUBE_PING cannot read deployment from
>>>>> json describing pod.
>>>>> There is already a bug on KUBE_PING for it:
>>>>> https://github.com/jgroups-extras/jgroups-kubernetes/issues/50
>>>>>
>>>>>
>>>>> пн, 8 июл. 2019 г. в 10:25, Sebastian Laskawiec <slaskawi at redhat.com>:
>>>>>
>>>>>> Ok, I'm glad it worked.
>>>>>>
>>>>>> Just for the future - with KUBE_PING, there's an additional option:
>>>>>> "split_clusters_during_rolling_update" set to true. See the documentation:
>>>>>> https://github.com/jgroups-extras/jgroups-kubernetes#kube_ping-configuration
>>>>>>
>>>>>> On Wed, Jul 3, 2019 at 4:32 PM Мартынов Илья <imartynovsp at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Sebastian,
>>>>>>> I use Kube_ping JGroups discovery plugin, it seems to be more
>>>>>>> reliable than IP Multicasting.
>>>>>>> Finally I've utilized choise #1 simply changing Strategy of
>>>>>>> Deployment in kubernetes from default "RollingUpdate" to "Recreate".
>>>>>>> Recreate does exactly what we need, first drops all existing pods and after
>>>>>>> that creates pods with new version.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ср, 3 июл. 2019 г. в 15:15, Sebastian Laskawiec <slaskawi at redhat.com
>>>>>>> >:
>>>>>>>
>>>>>>>> If you're using standalone-ha.xml without any extra parameters,
>>>>>>>> you're using UDP JGroups stack with IP Multicasting discovery. You also
>>>>>>>> suggested that you're using Pods, so I'm assuming you're using Kubernetes
>>>>>>>> with Flannel network plugin (as far as I know, other plugins do not support
>>>>>>>> IP Multicasting out of the box).
>>>>>>>>
>>>>>>>> So effectively you have 2 choices:
>>>>>>>> - Turn everything off, do the migration and start it again. Note,
>>>>>>>> that you need to shut everything down (this is not a rolling update
>>>>>>>> procedure).
>>>>>>>> - Reconfigure JGroups not to join the same cluster. The easiest
>>>>>>>> thing to do here is to modify the cluster attribute and make sure it's
>>>>>>>> different for the old and new cluster.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Sebastian
>>>>>>>>
>>>>>>>> On Wed, Jul 3, 2019 at 11:10 AM Мартынов Илья <
>>>>>>>> imartynovsp at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Just another question. Actually, I don't even need zero-downtime
>>>>>>>>> upgrade, I
>>>>>>>>> need just any upgrade. Now upgrade is blocked because new pod
>>>>>>>>> cannot start.
>>>>>>>>> Can we somehow make new pod to ignore non-readable messages from
>>>>>>>>> old pod
>>>>>>>>> and continue loading?
>>>>>>>>> Yes, session state won't be replicated, but it's more automated
>>>>>>>>> way then
>>>>>>>>> manual stop of old pod and starting a new pod after that.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ср, 3 июл. 2019 г. в 09:39, Мартынов Илья <imartynovsp at gmail.com>:
>>>>>>>>>
>>>>>>>>> > Hi Marek,
>>>>>>>>> >
>>>>>>>>> > Ok, I got it, thank you for response!
>>>>>>>>> >
>>>>>>>>> > вт, 2 июля 2019 г., 19:25 Marek Posolda <mposolda at redhat.com>:
>>>>>>>>> >
>>>>>>>>> >> I think you can't mix old and new Keycloak servers in same
>>>>>>>>> cluster. And
>>>>>>>>> >> rolling upgrade (zero downtime upgrade) is not yet supported.
>>>>>>>>> We plan to
>>>>>>>>> >> add support for it, but it won't be in a very near future as it
>>>>>>>>> will
>>>>>>>>> >> likely require quite a lot of work...
>>>>>>>>> >>
>>>>>>>>> >> In shortcut, it will be recommended to stop all pods with 4.5.0
>>>>>>>>> and then
>>>>>>>>> >> start pods with 6.0.1.
>>>>>>>>> >>
>>>>>>>>> >> Marek
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> On 02/07/2019 16:25, Мартынов Илья wrote:
>>>>>>>>> >> > Hello!
>>>>>>>>> >> > I have Keycloak 4.5.0.Final deployed in standalone-ha
>>>>>>>>> configuration in
>>>>>>>>> >> k8s
>>>>>>>>> >> > cluster. When I try to update Keycloak to version 6.0.1, the
>>>>>>>>> following
>>>>>>>>> >> > happens:
>>>>>>>>> >> > 1. K8s starts new pod with version 6.0.1
>>>>>>>>> >> > 2. Old pod still running, it will be terminated on
>>>>>>>>> successfull readiness
>>>>>>>>> >> > probe of the new pod
>>>>>>>>> >> > 3. New pod discovers old pod via JGroups, cache
>>>>>>>>> synchronization started
>>>>>>>>> >> > 4. Exception in new pod:
>>>>>>>>> >> > 02-07-2019;13:34:29,220 WARN
>>>>>>>>> [stateTransferExecutor-thread--p25-t1]
>>>>>>>>> >> > org.infinispan.statetransfer.InboundTransferTask ISPN000210:
>>>>>>>>> Failed to
>>>>>>>>> >> > request state of cache work from node idp-6569c544b
>>>>>>>>> >> > -hsd6g, segments {0-255}:
>>>>>>>>> org.infinispan.remoting.RemoteException:
>>>>>>>>> >> > ISPN000217: Received exception from idp-6569c544b-hsd6g, see
>>>>>>>>> cause for
>>>>>>>>> >> > remote stack trace
>>>>>>>>> >> > at org.infinispan at 9.4.8.Final
>>>>>>>>> >> >
>>>>>>>>> >>
>>>>>>>>> //org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:28)
>>>>>>>>> >> > ...
>>>>>>>>> >> > Caused by: java.io.IOException: Unknown type: 132
>>>>>>>>> >> > at
>>>>>>>>> >> >
>>>>>>>>> >>
>>>>>>>>> org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:681)
>>>>>>>>> >> > at
>>>>>>>>> >> >
>>>>>>>>> >>
>>>>>>>>> org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:355)
>>>>>>>>> >> > at
>>>>>>>>> >> >
>>>>>>>>> >>
>>>>>>>>> org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40)
>>>>>>>>> >> >
>>>>>>>>> >> > Looks like this exception blocks further Keycloak startup,
>>>>>>>>> because
>>>>>>>>> >> nothing
>>>>>>>>> >> > happens in logs afterwards. Also, my rest service deployed as
>>>>>>>>> JAX-RS
>>>>>>>>> >> bean
>>>>>>>>> >> > also doesn't respond, so pod is not treated as alive by
>>>>>>>>> Kubernetes.
>>>>>>>>> >> > Please help.
>>>>>>>>> >> > _______________________________________________
>>>>>>>>> >> > keycloak-dev mailing list
>>>>>>>>> >> > keycloak-dev at lists.jboss.org
>>>>>>>>> >> > https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> _______________________________________________
>>>>>>>>> keycloak-dev mailing list
>>>>>>>>> keycloak-dev at lists.jboss.org
>>>>>>>>> https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>>>
>>>>>>>>
More information about the keycloak-dev
mailing list