Thanks a lot for the patch. Let's move the conversation to the Pull Request.
On Mon, Jul 8, 2019 at 5:47 PM Мартынов Илья <imartynovsp(a)gmail.com> wrote:
Created
https://github.com/jgroups-extras/jgroups-kubernetes/pull/72
-
please take a look!
Not clear for me who and how will modify KUBE_PING jboss module, it
requires new dependecy now for JsonPath processing.
пн, 8 июл. 2019 г. в 17:31, Мартынов Илья <imartynovsp(a)gmail.com>:
> I've estimated the support of ReplicaSets and it looks like possible to
> implement in 1 day. So yes, I start working on it.
>
> пн, 8 июл. 2019 г. в 15:28, Sebastian Laskawiec <slaskawi(a)redhat.com>:
>
>> oooh right, I guess you hit this issue:
>>
https://github.com/jgroups-extras/jgroups-kubernetes/issues/50
>>
>> Sorry to hear that. Perhaps you'd be interested in contributing a fix to
>> KUBE_PING?
>>
>> On Mon, Jul 8, 2019 at 1:35 PM Мартынов Илья <imartynovsp(a)gmail.com>
>> wrote:
>>
>>> I've tried split_clusters_during_rolling_update option, but KUBE_PING
>>> start failing with NPE:
>>> Caused by: java.lang.NullPointerException
>>> at java.util.Objects.requireNonNull(Objects.java:203)
>>> at java.util.Optional.<init>(Optional.java:96)
>>> at java.util.Optional.of(Optional.java:108)
>>> at java.util.stream.FindOps$FindSink$OfRef.get(FindOps.java:193)
>>> at java.util.stream.FindOps$FindSink$OfRef.get(FindOps.java:190)
>>> at
>>> java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
>>> at
>>> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
>>> at
>>> java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464)
>>> at
>>> org.jgroups.protocols.kubernetes.KUBE_PING.findMembers(KUBE_PING.java:240)
>>> at
>>> org.jgroups.protocols.Discovery.findMembers(Discovery.java:211)
>>> at org.jgroups.protocols.Discovery.down(Discovery.java:350)
>>>
>>> In debug I see code in KUBE_PING that tries to find pod's deployment
>>> String senderParentDeployment = hosts.stream()
>>> .filter(pod -> senderIp.contains(pod.getIp()))
>>> .map(Pod::getParentDeployment)
>>> .findFirst().orElse(null);
>>> This code fails with NPE, as Pod::getParentDeployment returns null.
>>> Parent deployment not set because KUBE_PING cannot read deployment from
>>> json describing pod.
>>> There is already a bug on KUBE_PING for it:
>>>
https://github.com/jgroups-extras/jgroups-kubernetes/issues/50
>>>
>>>
>>> пн, 8 июл. 2019 г. в 10:25, Sebastian Laskawiec <slaskawi(a)redhat.com>:
>>>
>>>> Ok, I'm glad it worked.
>>>>
>>>> Just for the future - with KUBE_PING, there's an additional option:
>>>> "split_clusters_during_rolling_update" set to true. See the
documentation:
>>>>
https://github.com/jgroups-extras/jgroups-kubernetes#kube_ping-configuration
>>>>
>>>> On Wed, Jul 3, 2019 at 4:32 PM Мартынов Илья
<imartynovsp(a)gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks Sebastian,
>>>>> I use Kube_ping JGroups discovery plugin, it seems to be more
>>>>> reliable than IP Multicasting.
>>>>> Finally I've utilized choise #1 simply changing Strategy of
>>>>> Deployment in kubernetes from default "RollingUpdate" to
"Recreate".
>>>>> Recreate does exactly what we need, first drops all existing pods and
after
>>>>> that creates pods with new version.
>>>>>
>>>>>
>>>>>
>>>>> ср, 3 июл. 2019 г. в 15:15, Sebastian Laskawiec
<slaskawi(a)redhat.com
>>>>> >:
>>>>>
>>>>>> If you're using standalone-ha.xml without any extra
parameters,
>>>>>> you're using UDP JGroups stack with IP Multicasting
discovery. You also
>>>>>> suggested that you're using Pods, so I'm assuming
you're using Kubernetes
>>>>>> with Flannel network plugin (as far as I know, other plugins do
not support
>>>>>> IP Multicasting out of the box).
>>>>>>
>>>>>> So effectively you have 2 choices:
>>>>>> - Turn everything off, do the migration and start it again.
Note,
>>>>>> that you need to shut everything down (this is not a rolling
update
>>>>>> procedure).
>>>>>> - Reconfigure JGroups not to join the same cluster. The easiest
>>>>>> thing to do here is to modify the cluster attribute and make sure
it's
>>>>>> different for the old and new cluster.
>>>>>>
>>>>>> Thanks,
>>>>>> Sebastian
>>>>>>
>>>>>> On Wed, Jul 3, 2019 at 11:10 AM Мартынов Илья
<imartynovsp(a)gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Just another question. Actually, I don't even need
zero-downtime
>>>>>>> upgrade, I
>>>>>>> need just any upgrade. Now upgrade is blocked because new
pod
>>>>>>> cannot start.
>>>>>>> Can we somehow make new pod to ignore non-readable messages
from
>>>>>>> old pod
>>>>>>> and continue loading?
>>>>>>> Yes, session state won't be replicated, but it's more
automated way
>>>>>>> then
>>>>>>> manual stop of old pod and starting a new pod after that.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ср, 3 июл. 2019 г. в 09:39, Мартынов Илья
<imartynovsp(a)gmail.com>:
>>>>>>>
>>>>>>> > Hi Marek,
>>>>>>> >
>>>>>>> > Ok, I got it, thank you for response!
>>>>>>> >
>>>>>>> > вт, 2 июля 2019 г., 19:25 Marek Posolda
<mposolda(a)redhat.com>:
>>>>>>> >
>>>>>>> >> I think you can't mix old and new Keycloak
servers in same
>>>>>>> cluster. And
>>>>>>> >> rolling upgrade (zero downtime upgrade) is not yet
supported. We
>>>>>>> plan to
>>>>>>> >> add support for it, but it won't be in a very
near future as it
>>>>>>> will
>>>>>>> >> likely require quite a lot of work...
>>>>>>> >>
>>>>>>> >> In shortcut, it will be recommended to stop all pods
with 4.5.0
>>>>>>> and then
>>>>>>> >> start pods with 6.0.1.
>>>>>>> >>
>>>>>>> >> Marek
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> On 02/07/2019 16:25, Мартынов Илья wrote:
>>>>>>> >> > Hello!
>>>>>>> >> > I have Keycloak 4.5.0.Final deployed in
standalone-ha
>>>>>>> configuration in
>>>>>>> >> k8s
>>>>>>> >> > cluster. When I try to update Keycloak to
version 6.0.1, the
>>>>>>> following
>>>>>>> >> > happens:
>>>>>>> >> > 1. K8s starts new pod with version 6.0.1
>>>>>>> >> > 2. Old pod still running, it will be terminated
on successfull
>>>>>>> readiness
>>>>>>> >> > probe of the new pod
>>>>>>> >> > 3. New pod discovers old pod via JGroups,
cache
>>>>>>> synchronization started
>>>>>>> >> > 4. Exception in new pod:
>>>>>>> >> > 02-07-2019;13:34:29,220 WARN
>>>>>>> [stateTransferExecutor-thread--p25-t1]
>>>>>>> >> >
org.infinispan.statetransfer.InboundTransferTask ISPN000210:
>>>>>>> Failed to
>>>>>>> >> > request state of cache work from node
idp-6569c544b
>>>>>>> >> > -hsd6g, segments {0-255}:
>>>>>>> org.infinispan.remoting.RemoteException:
>>>>>>> >> > ISPN000217: Received exception from
idp-6569c544b-hsd6g, see
>>>>>>> cause for
>>>>>>> >> > remote stack trace
>>>>>>> >> > at org.infinispan(a)9.4.8.Final
>>>>>>> >> >
>>>>>>> >>
>>>>>>>
//org.infinispan.remoting.transport.ResponseCollectors.wrapRemoteException(ResponseCollectors.java:28)
>>>>>>> >> > ...
>>>>>>> >> > Caused by: java.io.IOException: Unknown type:
132
>>>>>>> >> > at
>>>>>>> >> >
>>>>>>> >>
>>>>>>>
org.infinispan.marshall.core.GlobalMarshaller.readNonNullableObject(GlobalMarshaller.java:681)
>>>>>>> >> > at
>>>>>>> >> >
>>>>>>> >>
>>>>>>>
org.infinispan.marshall.core.GlobalMarshaller.readNullableObject(GlobalMarshaller.java:355)
>>>>>>> >> > at
>>>>>>> >> >
>>>>>>> >>
>>>>>>>
org.infinispan.marshall.core.BytesObjectInput.readObject(BytesObjectInput.java:40)
>>>>>>> >> >
>>>>>>> >> > Looks like this exception blocks further
Keycloak startup,
>>>>>>> because
>>>>>>> >> nothing
>>>>>>> >> > happens in logs afterwards. Also, my rest
service deployed as
>>>>>>> JAX-RS
>>>>>>> >> bean
>>>>>>> >> > also doesn't respond, so pod is not treated
as alive by
>>>>>>> Kubernetes.
>>>>>>> >> > Please help.
>>>>>>> >> >
_______________________________________________
>>>>>>> >> > keycloak-dev mailing list
>>>>>>> >> > keycloak-dev(a)lists.jboss.org
>>>>>>> >> >
https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> _______________________________________________
>>>>>>> keycloak-dev mailing list
>>>>>>> keycloak-dev(a)lists.jboss.org
>>>>>>>
https://lists.jboss.org/mailman/listinfo/keycloak-dev
>>>>>>
>>>>>>