[Hawkular-dev] Release cadence

Wed Jul 22 02:52:12 EDT 2015

On 07/21/2015 03:50 PM, Stefan Negrea wrote:
> Hello,
>
> There are two issues that get interlaced here and should not be. One is to release individual components on a schedule, and second is making sure that all the components integrate nicely. Your proposal with "at least one release per week" is trying to fix a fix of a fix of previous decision. And in this mix, the two concepts of release and integration get so interlocked that we cannot make heads and tail.
>
> Here is the progression of things:
>
> Problem 1: monolithic code is bad, we need to componentize everything. Solution: create single purpose repositories, which resulted in about 20 repositories.
>
> Problem 2: how do we integrate all those components? Solution: create a Hawkular repository that depends on all the other repositories.
>
> Problem 3: integrated project idea works, but how do we really really integrate the code? Solution: publish changes in subcomponents as soon as possible.

The original idea of splitting the code was to separate the concerns of 
some parts of the code, split the relatively large team/project and 
provide public APIs that can be consumed by the Hawkular repository. 
Those components were supposed to be *libraries*. Splitting the code 
like this adds a *little* extra in release but adds benefits in quality 
and separation of concerns.

With splitted repositories you should force yourself about thinking well 
about the API and (usually) less subject to pick from private APIs, 
(Accidental) dependencies between various parts of the code are also 
more obvious.

This is where the problem raised...

So there are various mistakes IMO in the current organization:
	- Components as libraries suddenly became servers on their own
	- Components depends too much on each other, while Hawkular should be 
where most of the dependencies happen. (Your Pb6)
	- Most efforts after that (branching, SNAPSHOT...) are done toward 
recreating the RHQ single-repo experience.

Now we pay the extra without the benefits.

Again, my wish is that components are really independent and all the 
glue happens in Hawkular. (I would consider putting the bus in there)

And for the release cadence, I agree that we are in fast-changing mode 
so it's a bit more challenging. It will need to change soon as the API 
will need to remain stable and need less new features.

So it is important that as soon as a new API or API change is ready to 
be consumed or simply asked by someone who consumes a component there is 
a release done (name doesn't matter or original component cadence neither).

And to be clear again, SNAPSHOT is not the solution, SNAPSHOT are made 
to go back to single-repo style, if we need to get back to single-repo 
style, then let's have a single repo and not fake otherwise.

Same if we have some integration branch with SNAPSHOTs that we rush 
merged at the last minute in master, it just cheats the process...

Thomas

>
> Problem 4: publishing changes in components takes a long time, can we expedite the process and get changes even faster to integrated project? Solution: automate SNAPSHOT publication so you have freshly build binaries on almost every change.
>
> Problem 5: We need consistent builds for the integrated project. Solution: master branch in the integrated project only depends on released version of components, feature branches are short lived.
>
> Problem 6: It is almost impossible to align all the release of components and give enough runaway for the integrated project to ingest all the release subcomponets, especially since there inter-dependencies on the sub-projects themselves. Solution: publish sub-components officially at least once per week.
>
> Problem 7: There are major changes that are needed right-away in the integrated project. Solution: officially publish components more often than once per week. Publish as soon as the features makes it in. In fact, let's publish Alphas (Alphaxx) with every single change.
>
> Problem 8: Too many releases, it takes too much time to administer the process of releasing. Solution: Automate the release dependency management, emulate the SNAPSHOT injection concept but with Alpha "moniker".
>
> Problem 9: It is a nightmare to maintain on which version to depend on the integrate project or components that depend on other components. Solution: ??
>
> Problem 10: It is almost impossible to trace back the code that code that goes into the integrated project because there are so many Alphas. Solution: ???
>
>
> We are at level 5; from 6 forward I see them coming, just as I saw the rest of 5. At what point do we stop and say "wait a second! what we doing here? can we simplify all this??" From my perspective, we are arbitrarily setting requirements just to complicate things. There is always a happy medium and there is always ways to simplify. I would much rather think for 2 months on how to simplify things and do it once, then stack problems that make our lives harder for no reason.
>
>
> That being said, I am not against implementing your proposed solution in Hawkular Metrics. But I see two possible paths. One, escalate and resolve problems 6, and 7 at the same time with automation. That is primarily because we are so thin on resources and so stretched that we do not have time to not automate this. Two, get a release engineer (or another engineer) allocated to the team that will take care of these releases. If we are to proceed with your current proposal please let us know which one of these paths do you want Hawkular Metrics to take.
>
>
> At the same time, we can take the reversed path; rather than escalate problems, remove them from that stack. Why not look to simplify everything, revert a few problems and try to improve our productivity. We can revert to the use of SNAPSHOTS. In what capacity? Let's find the path that gives us the most benefits with the least amount of work.
>
>
> Thank you,
> Stefan
>
> ----- Original Message -----
>> From: "Heiko W.Rupp" <hrupp at redhat.com>
>> To: "Discussions around Hawkular development" <hawkular-dev at lists.jboss.org>
>> Sent: Tuesday, July 21, 2015 4:38:58 AM
>> Subject: [Hawkular-dev] Release cadence
>>
>> Hey
>>
>> I have observed that our current Hawkular cadence of 4 weeks
>> with similar cadences of components makes us end up with
>> long living integration branches and a larger rush near the
>> end to integrate them, get them for the first time tested in CI
>> and even for the first time tested in real world.
>>
>> In one of the last releases there was a changed implementation
>> in one component, that basically turned out as a no-op and
>> still returned a "200 OK" code, so clients thought everything is
>> happy, but it was not. We found the issue (through ppl looking
>> at the UI) and solved it, but it was in a rush.
>>
>> This certainly goes against all the ideas of "release early, release
>> often", "cut small slices", "changes go into CI/CD and go live quickly".
>> Remember the coin flipping ?
>>
>> Ideally we would always be able to integrate changes from
>> components into Hawkular (main), but I understand that with the
>> way maven and its release process to central works, it is also not
>> ideal to release many versions per day.
>>
>> With all of the above in mind, I propose that we move to a
>> "at least once per week" model, where we do a component release
>> at least once per week(*), which then in the four week stream form
>> a new Alpha release. The smaller releases do not need release notes,
>> I don't care if we use the micro number or a .AlphaY designator on
>> them, but they should be a release, that is not a (named) snapshot.
>> This will allow us to still have less efforts to do releases, but
>> keep being (more) agile and have earlier integrations and thus
>> less long living integration branches.
>>
>> On top of that, we need to provide new and/or changed apis(**)
>> early on in the 4 weeks cadence so that other components can
>> already start calling them, even if they are not yet functionally
>> complete.
>>
>> *) Of course only if a change to the component has been made.
>> **) Ideally with changed apis, we keep the old version around for
>>    a bit and offer the new version on top. Remember, that especially
>>    with non-compiletime bindings, we can not know which client is
>>    at what api version.
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>