10:48 a.m.
Hey,
on Kubernetes a common use case (that has been demonstrated by Thomas
and Juca recently) are rolling upgrades (blue/green, canary, ...) where a
new version of a service comes into play, gets scaled up and when this
is scaled up, the old version of the service gets scaled down and eventually
disappears.
It can also happen though that the new version is buggy and the previous
version gets scaled up again to take the full load until a fix is available.
We need to make sure to support those rolling upgrades and also the
rollback to the previous version of a service.
E.g. if a (No)Sql schema update happens in version N+1, a still
running instance of version N must not break by it, when N+1
is rolling out while N is still active. Similarly when N+1 has upgraded
the schema and the application gets rolled back to version N, it must
still be able to cope with the upgraded schema version.
Same applies to APIs, but this is something we have discussed and
mastered in the past.
We should perhaps investigate how we can automatically test
this scenario in our Travis testing.
4:54 p.m.
For APIs, I think this is possible.
But how do we handle the situations where we want to go from Cassandra 3.x to Cassandra
4.x? Or what happens when we need to update components when those components themselves
don't support this type of rollback?
----- Original Message -----
From: "Heiko W.Rupp" <hrupp(a)redhat.com>
To: "Discussions around Hawkular development"
<hawkular-dev(a)lists.jboss.org>
Sent: Friday, 3 March, 2017 4:48:41 AM
Subject: [Hawkular-dev] Rolling upgrades (on Kubernetes)
Hey,
on Kubernetes a common use case (that has been demonstrated by Thomas
and Juca recently) are rolling upgrades (blue/green, canary, ...) where a
new version of a service comes into play, gets scaled up and when this
is scaled up, the old version of the service gets scaled down and eventually
disappears.
I believe this only one type of scenario, there are other options depending on how your
application should behave.
For Cassandra, we want to bring down all the instances and bring up all new versions
(which is what we currently do in origin-metrics). The reason being that we want to use
the persistent storage from the previous versions. Otherwise we would need to bring up new
versions with empty volumes and then wait for the data to transfer from the old versions
to the new ones, which is really not what we want.
It can also happen though that the new version is buggy and the
previous
version gets scaled up again to take the full load until a fix is available.
We need to make sure to support those rolling upgrades and also the
rollback to the previous version of a service.
E.g. if a (No)Sql schema update happens in version N+1, a still
running instance of version N must not break by it, when N+1
is rolling out while N is still active. Similarly when N+1 has upgraded
the schema and the application gets rolled back to version N, it must
still be able to cope with the upgraded schema version.
Same applies to APIs, but this is something we have discussed and
mastered in the past.
We should perhaps investigate how we can automatically test
this scenario in our Travis testing.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
5:54 p.m.
On 3 Mar 2017, at 16:54, Matt Wringe wrote:
But how do we handle the situations where we want to go from
Cassandra
3.x to Cassandra 4.x? Or what happens when we need to update
components when those components themselves don't support this type of
rollback?
I did not say this is easy :)
Let me construct a case to illustrate this.
In version N we have a column that stores a string in the form
"tenantId:metricid". Now we decide for N+1 that this is not
good and we need to split that in two separate columns.
In a traditional way we would do the split and also remove the
old single column. Here a rollback is not possible.
If we change this to keep the old column around and the two
new ones we are safe for version N+1, but as there may be
still version N containers around we may end up with inconsistencies
when version N updates an item and N+1 already has split that row.
Here version N+1 would potentially need to read/write from both
scenarios and also update both (if the old one still exists, see below).
This allows rolling back.
Version N+2 could then drop the old "tenantId:metricid" column
and only operate on the split columns.
Heavier updates like C* version 3.x to 4.x can probably really not
be done without downtime - especially when they require massive
migrations of Gigabytes of data.
But still in this case the Metrics frontend code could work against
both versions of C* and redundantly store data in both while slowly
migrating data over. Reads could then still go to the old code by
default until the migration is complete.
8:18 p.m.
On Mar 3, 2017, at 11:54 AM, Heiko W.Rupp <hrupp(a)redhat.com>
wrote:
On 3 Mar 2017, at 16:54, Matt Wringe wrote:
> But how do we handle the situations where we want to go from Cassandra
> 3.x to Cassandra 4.x? Or what happens when we need to update
> components when those components themselves don't support this type of
> rollback?
I did not say this is easy :)
Let me construct a case to illustrate this.
In version N we have a column that stores a string in the form
"tenantId:metricid". Now we decide for N+1 that this is not
good and we need to split that in two separate columns.
In a traditional way we would do the split and also remove the
old single column. Here a rollback is not possible.
If we change this to keep the old column around and the two
new ones we are safe for version N+1, but as there may be
still version N containers around we may end up with inconsistencies
when version N updates an item and N+1 already has split that row.
Here version N+1 would potentially need to read/write from both
scenarios and also update both (if the old one still exists, see below).
This allows rolling back.
Version N+2 could then drop the old "tenantId:metricid" column
and only operate on the split columns.
Heavier updates like C* version 3.x to 4.x can probably really not
be done without downtime - especially when they require massive
migrations of Gigabytes of data.
But still in this case the Metrics frontend code could work against
both versions of C* and redundantly store data in both while slowly
migrating data over. Reads could then still go to the old code by
default until the migration is complete.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
<https://lists.jboss.org/mailman/listinfo/hawkular-dev>
As far as I know rolling upgrades of Cassandra with a major version change is well
supported, and rolling upgrades is pretty common among Cassandra users. Here is the
scenario I am not sure we could handle without downtime. We want to make make a schema
change that involves something introduced in the new version of Cassandra, e.g., SASI
index.
9:29 p.m.
----- Original Message -----
From: "John Sanda" <jsanda(a)redhat.com>
To: "Discussions around Hawkular development"
<hawkular-dev(a)lists.jboss.org>
Sent: Friday, 3 March, 2017 2:18:39 PM
Subject: Re: [Hawkular-dev] Rolling upgrades (on Kubernetes)
On Mar 3, 2017, at 11:54 AM, Heiko W.Rupp < hrupp(a)redhat.com > wrote:
On 3 Mar 2017, at 16:54, Matt Wringe wrote:
But how do we handle the situations where we want to go from Cassandra
3.x to Cassandra 4.x? Or what happens when we need to update
components when those components themselves don't support this type of
rollback?
I did not say this is easy :)
Let me construct a case to illustrate this.
In version N we have a column that stores a string in the form
"tenantId:metricid". Now we decide for N+1 that this is not
good and we need to split that in two separate columns.
In a traditional way we would do the split and also remove the
old single column. Here a rollback is not possible.
If we change this to keep the old column around and the two
new ones we are safe for version N+1, but as there may be
still version N containers around we may end up with inconsistencies
when version N updates an item and N+1 already has split that row.
Here version N+1 would potentially need to read/write from both
scenarios and also update both (if the old one still exists, see below).
This allows rolling back.
Version N+2 could then drop the old "tenantId:metricid" column
and only operate on the split columns.
Heavier updates like C* version 3.x to 4.x can probably really not
be done without downtime - especially when they require massive
migrations of Gigabytes of data.
But still in this case the Metrics frontend code could work against
both versions of C* and redundantly store data in both while slowly
migrating data over. Reads could then still go to the old code by
default until the migration is complete.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
As far as I know rolling upgrades of Cassandra with a major version change is
well supported, and rolling upgrades is pretty common among Cassandra users.
Here is the scenario I am not sure we could handle without downtime. We want
to make make a schema change that involves something introduced in the new
version of Cassandra, e.g., SASI index.
Yeah, its not the rolling upgrades I think we will run into problems with, its the
downgrade when someone wants to revert back to the previous version that might cause
problems.
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
2610
days inactive
2610
days old
4 comments
3 participants
participants (3)
-
Heiko W.Rupp -
John Sanda -
Matt Wringe