I've tried split_clusters_during_rolling_update option, but KUBE_PING start
failing with NPE:
Caused by: java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at java.util.Optional.<init>(Optional.java:96)
at java.util.Optional.of(Optional.java:108)
at java.util.stream.FindOps$FindSink$OfRef.get(FindOps.java:193)
at java.util.stream.FindOps$FindSink$OfRef.get(FindOps.java:190)
at
java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464)
at
org.jgroups.protocols.kubernetes.KUBE_PING.findMembers(KUBE_PING.java:240)
at org.jgroups.protocols.Discovery.findMembers(Discovery.java:211)
at org.jgroups.protocols.Discovery.down(Discovery.java:350)
In debug I see code in KUBE_PING that tries to find pod's deployment
String senderParentDeployment = hosts.stream()
.filter(pod -> senderIp.contains(pod.getIp()))
.map(Pod::getParentDeployment)
.findFirst().orElse(null);
This code fails with NPE, as Pod::getParentDeployment returns null. Parent
deployment not set because KUBE_PING cannot read deployment from json
describing pod.
There is already a bug on KUBE_PING for it:
пн, 8 июл. 2019 г. в 10:25, Sebastian Laskawiec <slaskawi(a)redhat.com>:
Ok, I'm glad it worked.
Just for the future - with KUBE_PING, there's an additional option:
"split_clusters_during_rolling_update" set to true. See the documentation:
https://github.com/jgroups-extras/jgroups-kubernetes#kube_ping-configuration
On Wed, Jul 3, 2019 at 4:32 PM Мартынов Илья <imartynovsp(a)gmail.com>
wrote:
> Thanks Sebastian,
> I use Kube_ping JGroups discovery plugin, it seems to be more reliable
> than IP Multicasting.
> Finally I've utilized choise #1 simply changing Strategy of Deployment in
> kubernetes from default "RollingUpdate" to "Recreate". Recreate
does
> exactly what we need, first drops all existing pods and after that creates
> pods with new version.
>
>
>
> ср, 3 июл. 2019 г. в 15:15, Sebastian Laskawiec <slaskawi(a)redhat.com>:
>
>> If you're using standalone-ha.xml without any extra parameters, you're
>> using UDP JGroups stack with IP Multicasting discovery. You also suggested
>> that you're using Pods, so I'm assuming you're using Kubernetes with
>> Flannel network plugin (as far as I know, other plugins do not support IP
>> Multicasting out of the box).
>>
>> So effectively you have 2 choices:
>> - Turn everything off, do the migration and start it again. Note, that
>> you need to shut everything down (this is not a rolling update procedure).
>> - Reconfigure JGroups not to join the same cluster. The easiest thing to
>> do here is to modify the cluster attribute and make sure it's different for
>> the old and new cluster.
>>
>> Thanks,
>> Sebastian
>>
>> On Wed, Jul 3, 2019 at 11:10 AM Мартынов Илья <imartynovsp(a)gmail.com>
>> wrote:
>>
>>> Just another question. Actually, I don't even need zero-downtime
>>> upgrade, I
>>> need just any upgrade. Now upgrade is blocked because new pod cannot
>>> start.
>>> Can we somehow make new pod to ignore non-readable messages from old pod
>>> and continue loading?
>>> Yes, session state won't be replicated, but it's more automated way
then
>>> manual stop of old pod and starting a new pod after that.
>>>
>>>
>>>
>>> ср, 3 июл. 2019 г. в 09:39, Мартынов Илья <imartynovsp(a)gmail.com>:
>>>
>>> > Hi Marek,
>>> >
>>> > Ok, I got it, thank you for response!
>>> >
>>> > вт, 2 июля 2019 г., 19:25 Marek Posolda <mposolda(a)redhat.com>:
>>> >
>>> >> I think you can't mix old and new Keycloak servers in same
cluster.
>>> And
>>> >> rolling upgrade (zero downtime upgrade) is not yet supported. We
>>> plan to
>>> >> add support for it, but it won't be in a very near future as it
will
>>> >> likely require quite a lot of work...
>>> >>
>>> >> In shortcut, it will be recommended to stop all pods with 4.5.0 and
>>> then
>>> >> start pods with 6.0.1.
>>> >>
>>> >> Marek
>>> >>
>>> >>
>>> >> On 02/07/2019 16:25, Мартынов Илья wrote:
>>> >> > Hello!
>>> >> > I have Keycloak 4.5.0.Final deployed in standalone-ha
>>> configuration in
>>> >> k8s
>>> >> > cluster. When I try to update Keycloak to version 6.0.1, the
>>> following
>>> >> > happens:
>>> >> > 1. K8s starts new pod with version 6.0.1
>>> >> > 2. Old pod still running, it will be terminated on successfull
>>> readiness
>>> >> > probe of the new pod
>>> >> > 3. New pod discovers old pod via JGroups, cache
synchronization
>>> started
>>> >> > 4. Exception in new pod:
>>> >> > 02-07-2019;13:34:29,220 WARN
[stateTransferExecutor-thread--p25-t1]
>>> >> > org.infinispan.statetransfer.InboundTransferTask ISPN000210:
>>> Failed to
>>> >> > request state of cache work from node idp-6569c544b
>>> >> > -hsd6g, segments {0-255}:
org.infinispan.remoting.RemoteException:
>>> >> > ISPN000217: Received exception from idp-6569c544b-hsd6g, see
cause
>>> for
>>> >> > remote stack trace
>>> >> > at org.infinispan(a)9.4.8.Final
>>> >> >
>>> >>
>>>
//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:28)
>>> >> > ...
>>> >> > Caused by: java.io.IOException: Unknown type: 132
>>> >> > at
>>> >> >
>>> >>
>>>
org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:681)
>>> >> > at
>>> >> >
>>> >>
>>>
org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:355)
>>> >> > at
>>> >> >
>>> >>
>>>
org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40)
>>> >> >
>>> >> > Looks like this exception blocks further Keycloak startup,
because
>>> >> nothing
>>> >> > happens in logs afterwards. Also, my rest service deployed as
>>> JAX-RS
>>> >> bean
>>> >> > also doesn't respond, so pod is not treated as alive by
Kubernetes.
>>> >> > Please help.
>>> >> > _______________________________________________
>>> >> > keycloak-dev mailing list
>>> >> > keycloak-dev(a)lists.jboss.org
>>> >> >
https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>> >>
>>> >>
>>> >>
>>> _______________________________________________
>>> keycloak-dev mailing list
>>> keycloak-dev(a)lists.jboss.org
>>>
https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>
>>