9:56 a.m.
Hey all,
Infinispan has historically had two ways of performing live migration
between two clusters, via Hot Rod and via REST. We do not currently
provide an offline migration, although we do have a cache store
migration tool.
Gustavo has recently made several changes to the Hot Rod implementation
which have improved it greatly. The REST implementation is still not
robust enough, but I think we can abandon it and just focus on the Hot
Rod one ever for servers using REST.
The following is a list of stuff, mostly compiled by Gustavo, that needs
to be done to make everything smooth and robust:
1) Need a way to automate client redirection to the new cluster. I've
often referred to this as L4 client intelligence, which can also be used
for server-assisted cross-site failover.
2) Need a way to "rollback" the process in case of failures during the
migration: redirecting the clients back to the original cluster without
data loss. This would use the above L4 strategy.
3) Expose metrics and progress
4) Expose a way to cancel the process
5) Expose a container-wide migration process which can be applied to all
caches instead of one cache at a time.
6) The migration process should also take care of automatically
configuring the endpoints / remote cache stores at the beginning of the
process and removing any changes at the end.
6) Provide a future-proof format for the entries
7) Implement dump and restore capabilities which can export the contents
of a cluster to a file (compressed, encrypted, etc) or a collection of
files (one per cache).
Anything else ?
Tristan
--
Tristan Tarrant
Infinispan Lead
JBoss, a division of Red Hat
5:50 a.m.
On středa 17. května 2017 16:56:25 CEST Tristan Tarrant wrote:
2) Need a way to "rollback" the process in case of failures
during the
migration: redirecting the clients back to the original cluster without
data loss. This would use the above L4 strategy.
it's not only about redirecting clients - IIRC newly created entries on target
cluster are not propagated back to source cluster during rolling upgrade, so
we need also somehow sync these new data back to source cluster during the
rollback to avoid data losses. Same applies to "cancel process" feature
6:05 a.m.
On Fri, May 19, 2017 at 11:50 AM, Vojtech Juranek <vjuranek(a)redhat.com>
wrote:
On středa 17. května 2017 16:56:25 CEST Tristan Tarrant wrote:
> 2) Need a way to "rollback" the process in case of failures during the
> migration: redirecting the clients back to the original cluster without
> data loss. This would use the above L4 strategy.
it's not only about redirecting clients - IIRC newly created entries on
target
cluster are not propagated back to source cluster during rolling upgrade,
After the latest changes, new entries written to the target cluster are
supposed to propagated back to the source [1].
Did you find any issue with it?
[1] https://issues.jboss.org/browse/ISPN-7586
Thanks,
Gustavo
so
we need also somehow sync these new data back to source cluster during the
rollback to avoid data losses. Same applies to "cancel process" feature
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
1:45 a.m.
On pátek 19. května 2017 13:05:00 CEST Gustavo Fernandes wrote:
After the latest changes, new entries written to the target cluster
are
supposed to propagated back to the source [1].
I thought that when the rolling upgrade is in progress, entries written to
target cluster are not written to source cluster. If this is incorrect, what
actually does L70 in DistCacheWriterInterceptor [1]?
Thanks
[1] https://github.com/infinispan/infinispan/blob/master/core/src/main/java/
org/infinispan/interceptors/impl/DistCacheWriterInterceptor.java#L70
2:35 a.m.
On Mon, May 22, 2017 at 7:45 AM, Vojtech Juranek <vjuranek(a)redhat.com>
wrote:
On pátek 19. května 2017 13:05:00 CEST Gustavo Fernandes wrote:
> After the latest changes, new entries written to the target cluster are
> supposed to propagated back to the source [1].
I thought that when the rolling upgrade is in progress, entries written to
target cluster are not written to source cluster.
During a RU, there are two agents writing data to the target cluster: the
user and the RU process itself.
Since the remote store in the target cluster is not supposed to be used in
read-only mode anymore, all
data changes by the user in the target cluster during rolling upgrade are
propagate remotely to the source.
OTOH, data written by the Rolling Upgrader itself is not written back to
the source, since it is merely doing the migration.
If this is incorrect, what
actually does L70 in DistCacheWriterInterceptor [1]?
This line basically says: if data is being written by the rolling upgrade
process itself, write it to the stores (excluding the remoteStore),
regardless if the command is conditional or not.
Thanks
[1] https://github.com/infinispan/infinispan/blob/master/core/
src/main/java/
org/infinispan/interceptors/impl/DistCacheWriterInterceptor.java#L70
<https://github.com/infinispan/infinispan/blob/master/core/src/main/java/%...
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
6:17 a.m.
+1 for Vojtech
yes the client's need to moved to the new cluster in one shot current, that
was discussed before.
And it makes the migration because most of the customers are not able to
make that happen.
So there is a small possibility of inconsistence if clients connect to the
old server update entries until the new server already migrated it.
I see two options
1)
source server need to propagate active to target on update
2)
with the new L4 strategy all clients are moved automatically to the target.
So the source is not updated.
I only see a small possibility for this to happen during switch
- a client might still have a request to the source until other clients are
moved to target and already accessed the key
- a new client connects with old properties, here we need to ensure that
the first request is redirected to the target and not update the source
On Fri, May 19, 2017 at 12:50 PM, Vojtech Juranek <vjuranek(a)redhat.com>
wrote:
On středa 17. května 2017 16:56:25 CEST Tristan Tarrant wrote:
> 2) Need a way to "rollback" the process in case of failures during the
> migration: redirecting the clients back to the original cluster without
> data loss. This would use the above L4 strategy.
it's not only about redirecting clients - IIRC newly created entries on
target
cluster are not propagated back to source cluster during rolling upgrade,
so
we need also somehow sync these new data back to source cluster during the
rollback to avoid data losses. Same applies to "cancel process" feature
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
6:52 a.m.
On Fri, May 19, 2017 at 1:18 PM Wolf Fink <wfink(a)redhat.com> wrote:
+1 for Vojtech
yes the client's need to moved to the new cluster in one shot current,
that was discussed before.
And it makes the migration because most of the customers are not able to
make that happen.
So there is a small possibility of inconsistence if clients connect to the
old server update entries until the new server already migrated it.
I see two options
1)
source server need to propagate active to target on update
2)
with the new L4 strategy all clients are moved automatically to the
target. So the source is not updated.
I only see a small possibility for this to happen during switch
- a client might still have a request to the source until other clients
are moved to target and already accessed the key
- a new client connects with old properties, here we need to ensure that
the first request is redirected to the target and not update the source
Could you please tell me what L4 means in this context? Are you referring
to L4 routing/switching (transport level) or new Hot Rod client
intelligence?
In Kubernetes/OpenShift governing an Infinispan cluster by a Load Balancer
could do the trick. If all clients will use Service URL, once Kubernetes
kills all "old" Pods, all TCP socket connection will break and the client
will retry. This will result in massive load of error messages but the
client will eventually connect to the new cluster.
On Fri, May 19, 2017 at 12:50 PM, Vojtech Juranek <vjuranek(a)redhat.com>
wrote:
> On středa 17. května 2017 16:56:25 CEST Tristan Tarrant wrote:
> > 2) Need a way to "rollback" the process in case of failures during
the
> > migration: redirecting the clients back to the original cluster without
> > data loss. This would use the above L4 strategy.
>
> it's not only about redirecting clients - IIRC newly created entries on
> target
> cluster are not propagated back to source cluster during rolling upgrade,
> so
> we need also somehow sync these new data back to source cluster during the
> rollback to avoid data losses. Same applies to "cancel process" feature
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
SEBASTIAN ŁASKAWIEC
INFINISPAN DEVELOPER
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig>
11:59 a.m.
This is mentioned by Tristan "L4 client intelligence" which mean HotRod not
Network
On Mon, May 22, 2017 at 1:52 PM, Sebastian Laskawiec <slaskawi(a)redhat.com>
wrote:
On Fri, May 19, 2017 at 1:18 PM Wolf Fink <wfink(a)redhat.com> wrote:
> +1 for Vojtech
>
> yes the client's need to moved to the new cluster in one shot current,
> that was discussed before.
> And it makes the migration because most of the customers are not able to
> make that happen.
> So there is a small possibility of inconsistence if clients connect to
> the old server update entries until the new server already migrated it.
>
> I see two options
> 1)
> source server need to propagate active to target on update
> 2)
> with the new L4 strategy all clients are moved automatically to the
> target. So the source is not updated.
> I only see a small possibility for this to happen during switch
> - a client might still have a request to the source until other clients
> are moved to target and already accessed the key
> - a new client connects with old properties, here we need to ensure that
> the first request is redirected to the target and not update the source
>
Could you please tell me what L4 means in this context? Are you referring
to L4 routing/switching (transport level) or new Hot Rod client
intelligence?
In Kubernetes/OpenShift governing an Infinispan cluster by a Load Balancer
could do the trick. If all clients will use Service URL, once Kubernetes
kills all "old" Pods, all TCP socket connection will break and the client
will retry. This will result in massive load of error messages but the
client will eventually connect to the new cluster.
>
> On Fri, May 19, 2017 at 12:50 PM, Vojtech Juranek <vjuranek(a)redhat.com>
> wrote:
>
>> On středa 17. května 2017 16:56:25 CEST Tristan Tarrant wrote:
>> > 2) Need a way to "rollback" the process in case of failures during
the
>> > migration: redirecting the clients back to the original cluster without
>> > data loss. This would use the above L4 strategy.
>>
>> it's not only about redirecting clients - IIRC newly created entries on
>> target
>> cluster are not propagated back to source cluster during rolling
>> upgrade, so
>> we need also somehow sync these new data back to source cluster during
>> the
>> rollback to avoid data losses. Same applies to "cancel process"
feature
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
SEBASTIAN ŁASKAWIEC
INFINISPAN DEVELOPER
Red Hat EMEA <https://www.redhat.com/>
<https://red.ht/sig>
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
7:59 a.m.
Hi Wolf,
that was exactly my thought. Clients redirected to the target cluster do
not get updates written by other clients in the source cluster during
the rolling upgrade process. It is because the clients in target cluster
won't read the data through the remote cache store if they already have
the requested key in the local memory. Is there a BZ/JIRA for this?
Martin
On 19.5.2017 13:17, Wolf Fink wrote:
+1 for Vojtech
yes the client's need to moved to the new cluster in one shot current,
that was discussed before.
And it makes the migration because most of the customers are not able
to make that happen.
So there is a small possibility of inconsistence if clients connect to
the old server update entries until the new server already migrated it.
I see two options
1)
source server need to propagate active to target on update
2)
with the new L4 strategy all clients are moved automatically to the
target. So the source is not updated.
I only see a small possibility for this to happen during switch
- a client might still have a request to the source until other
clients are moved to target and already accessed the key
- a new client connects with old properties, here we need to ensure
that the first request is redirected to the target and not update the
source
On Fri, May 19, 2017 at 12:50 PM, Vojtech Juranek <vjuranek(a)redhat.com
<mailto:vjuranek@redhat.com>> wrote:
On středa 17. května 2017 16:56:25 CEST Tristan Tarrant wrote:
> 2) Need a way to "rollback" the process in case of failures
during the
> migration: redirecting the clients back to the original cluster
without
> data loss. This would use the above L4 strategy.
it's not only about redirecting clients - IIRC newly created
entries on target
cluster are not propagated back to source cluster during rolling
upgrade, so
we need also somehow sync these new data back to source cluster
during the
rollback to avoid data losses. Same applies to "cancel process"
feature
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org <mailto:infinispan-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/infinispan-dev
<https://lists.jboss.org/mailman/listinfo/infinispan-dev>
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
9:12 a.m.
On Mon, May 22, 2017 at 1:59 PM, Martin Gencur <mgencur(a)redhat.com> wrote:
It will follow soon. In the meantime, please make sure clients are pointing
to the new server.
Gustavo
> Martin
> On 19.5.2017 13:17, Wolf Fink
wrote:
> +1 for Vojtech
> yes the client's need to moved to the new cluster in
one shot current,
> that was discussed before.
> And it makes the migration because most of the customers are not able to
> make that happen.
> So there is a small possibility of inconsistence if clients connect to the
> old server update entries until the new server already migrated it.
> I see two options
> 1)
> source server need to propagate active to target on update
> 2)
> with the new L4 strategy all clients are moved automatically to the
> target. So the source is not updated.
> I only see a small possibility for this to happen during switch
> - a client might still have a request to the source until other clients
> are moved to target and already accessed the key
> - a new client connects with old properties, here we need to ensure that
> the first request is redirected to the target and not update the source
> On Fri, May 19, 2017 at 12:50 PM, Vojtech Juranek
<vjuranek(a)redhat.com > wrote:
>> On středa 17. května 2017 16:56:25 CEST Tristan
Tarrant wrote:
>> > 2) Need a way to "rollback" the process in case of failures during
the
>> > migration: redirecting the clients back to the original cluster without
>> > data loss. This would use the above L4 strategy.
> >> it's not only about redirecting clients - IIRC
newly created entries on
>> target
>> cluster are not propagated back to source cluster during rolling upgrade,
>> so
>> we need also somehow sync these new data back to source cluster during the
>> rollback to avoid data losses. Same applies to "cancel process"
feature
>> _______________________________________________
>> infinispan-dev mailing list
>> infinispan-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> > _______________________________________________
> infinispan-dev mailing
listinfinispan-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
Hi Wolf,
that was exactly my thought. Clients redirected to the target cluster do
not get updates written by other clients in the source cluster during the
rolling upgrade process. It is because the clients in target cluster won't
read the data through the remote cache store if they already have the
requested key in the local memory.
No, it won't, the source cluster is not supposed to be written to during
Rolling Upgrade. That's why the "L4" will prevent that. As per Wolf's
comments:
a client might still have a request to the source until other clients
are
moved to target and already accessed the key
The source cluster will enter a "redirect" mode. Every new client and new
operation will be sent to the new cluster.
Ongoing operations will need to be re-done in the new cluster.
a new client connects with old properties, here we need to ensure
that
the first request is redirected to the target and not update the source
The old server will be in "redirect" mode, this new client will be
redirected to the new cluster. After the RU completes, this new client will
not
be able to connect anymore since the old cluster will have been destroyed.
Is there a BZ/JIRA for this?
3057
days inactive
3062
days old
infinispan-dev@lists.jboss.org
10 comments
6 participants
participants (6)
-
Gustavo Fernandes -
Martin Gencur -
Sebastian Laskawiec -
Tristan Tarrant -
Vojtech Juranek -
Wolf Fink