From: "Heiko W.Rupp" <hrupp(a)redhat.com&gt; To: "Discussions around Hawkular development" <hawkular-dev(a)lists.jboss.org&gt; Sent: Tuesday, July 21, 2015 4:38:58 AM Subject: [Hawkular-dev] Release cadence Hey I have observed that our current Hawkular cadence of 4 weeks with similar cadences of components makes us end up with long living integration branches and a larger rush near the end to integrate them, get them for the first time tested in CI and even for the first time tested in real world. In one of the last releases there was a changed implementation in one component, that basically turned out as a no-op and still returned a "200 OK" code, so clients thought everything is happy, but it was not. We found the issue (through ppl looking at the UI) and solved it, but it was in a rush. This certainly goes against all the ideas of "release early, release often", "cut small slices", "changes go into CI/CD and go live quickly". Remember the coin flipping ? Ideally we would always be able to integrate changes from components into Hawkular (main), but I understand that with the way maven and its release process to central works, it is also not ideal to release many versions per day. With all of the above in mind, I propose that we move to a "at least once per week" model, where we do a component release at least once per week(*), which then in the four week stream form a new Alpha release. The smaller releases do not need release notes, I don't care if we use the micro number or a .AlphaY designator on them, but they should be a release, that is not a (named) snapshot. This will allow us to still have less efforts to do releases, but keep being (more) agile and have earlier integrations and thus less long living integration branches. On top of that, we need to provide new and/or changed apis(**) early on in the 4 weeks cadence so that other components can already start calling them, even if they are not yet functionally complete. *) Of course only if a change to the component has been made. **) Ideally with changed apis, we keep the old version around for a bit and offer the new version on top. Remember, that especially with non-compiletime bindings, we can not know which client is at what api version. _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

Each repo needs to have a "dev/latest" branch. Should the component need them, the pom.xml files in those dev/latest branches can refer to the latest-n-greatest snapshot versions of components (we can have travis publish the snapshot builds from these dev/latest branches). Therefore, if you want the latest-n-greatest to test integration, switch to dev/latest branches and build them locally. If you want a released version that never changes go to the master branch (which is the latest release) or go to one of the release tags and build from that. Once we are getting ready to prepare for a release, we create PRs to merge dev/latest to master, changing the poms to the release versions that are needed. How long between merging dev/latest to master and release? I dunno. If we do this right, we won't need a quick 1-week turnaround time. Every 2-weeks? 3-weeks? keep it a month? I don't know. Make something up. I vote every 17 days - only because its just as arbitrary as any other time period we might pick :) ----- Original Message ----- > Hello, > > There are two issues that get interlaced here and should not be. One is to > release individual components on a schedule, and second is making sure > that > all the components integrate nicely. Your proposal with "at least one > release per week" is trying to fix a fix of a fix of previous decision. > And > in this mix, the two concepts of release and integration get so > interlocked > that we cannot make heads and tail. > > Here is the progression of things: > > Problem 1: monolithic code is bad, we need to componentize everything. > Solution: create single purpose repositories, which resulted in about 20 > repositories. > > Problem 2: how do we integrate all those components? Solution: create a > Hawkular repository that depends on all the other repositories. > > Problem 3: integrated project idea works, but how do we really really > integrate the code? Solution: publish changes in subcomponents as soon as > possible. > > Problem 4: publishing changes in components takes a long time, can we > expedite the process and get changes even faster to integrated project? > Solution: automate SNAPSHOT publication so you have freshly build binaries > on almost every change. > > Problem 5: We need consistent builds for the integrated project. Solution: > master branch in the integrated project only depends on released version > of > components, feature branches are short lived. > > Problem 6: It is almost impossible to align all the release of components > and give enough runaway for the integrated project to ingest all the > release subcomponets, especially since there inter-dependencies on the > sub-projects themselves. Solution: publish sub-components officially at > least once per week. > > Problem 7: There are major changes that are needed right-away in the > integrated project. Solution: officially publish components more often > than > once per week. Publish as soon as the features makes it in. In fact, let's > publish Alphas (Alphaxx) with every single change. > > Problem 8: Too many releases, it takes too much time to administer the > process of releasing. Solution: Automate the release dependency > management, > emulate the SNAPSHOT injection concept but with Alpha "moniker". > > Problem 9: It is a nightmare to maintain on which version to depend on the > integrate project or components that depend on other components. Solution: > ?? > > Problem 10: It is almost impossible to trace back the code that code that > goes into the integrated project because there are so many Alphas. > Solution: ??? > > > We are at level 5; from 6 forward I see them coming, just as I saw the > rest > of 5. At what point do we stop and say "wait a second! what we doing here? > can we simplify all this??" From my perspective, we are arbitrarily > setting > requirements just to complicate things. There is always a happy medium and > there is always ways to simplify. I would much rather think for 2 months > on > how to simplify things and do it once, then stack problems that make our > lives harder for no reason. > > > That being said, I am not against implementing your proposed solution in > Hawkular Metrics. But I see two possible paths. One, escalate and resolve > problems 6, and 7 at the same time with automation. That is primarily > because we are so thin on resources and so stretched that we do not have > time to not automate this. Two, get a release engineer (or another > engineer) allocated to the team that will take care of these releases. If > we are to proceed with your current proposal please let us know which one > of these paths do you want Hawkular Metrics to take. > > > At the same time, we can take the reversed path; rather than escalate > problems, remove them from that stack. Why not look to simplify > everything, > revert a few problems and try to improve our productivity. We can revert > to > the use of SNAPSHOTS. In what capacity? Let's find the path that gives us > the most benefits with the least amount of work. > > > Thank you, > Stefan > > ----- Original Message ----- > > > From: "Heiko W.Rupp" <hrupp(a)redhat.com&gt; > > To: "Discussions around Hawkular development" > > <hawkular-dev(a)lists.jboss.org&gt; > > Sent: Tuesday, July 21, 2015 4:38:58 AM > > Subject: [Hawkular-dev] Release cadence > > > > Hey > > > > I have observed that our current Hawkular cadence of 4 weeks > > with similar cadences of components makes us end up with > > long living integration branches and a larger rush near the > > end to integrate them, get them for the first time tested in CI > > and even for the first time tested in real world. > > > > In one of the last releases there was a changed implementation > > in one component, that basically turned out as a no-op and > > still returned a "200 OK" code, so clients thought everything is > > happy, but it was not. We found the issue (through ppl looking > > at the UI) and solved it, but it was in a rush. > > > > This certainly goes against all the ideas of "release early, release > > often", "cut small slices", "changes go into CI/CD and go live quickly". > > Remember the coin flipping ? > > > > Ideally we would always be able to integrate changes from > > components into Hawkular (main), but I understand that with the > > way maven and its release process to central works, it is also not > > ideal to release many versions per day. > > > > With all of the above in mind, I propose that we move to a > > "at least once per week" model, where we do a component release > > at least once per week(*), which then in the four week stream form > > a new Alpha release. The smaller releases do not need release notes, > > I don't care if we use the micro number or a .AlphaY designator on > > them, but they should be a release, that is not a (named) snapshot. > > This will allow us to still have less efforts to do releases, but > > keep being (more) agile and have earlier integrations and thus > > less long living integration branches. > > > > On top of that, we need to provide new and/or changed apis(**) > > early on in the 4 weeks cadence so that other components can > > already start calling them, even if they are not yet functionally > > complete. > > > > *) Of course only if a change to the component has been made. > > **) Ideally with changed apis, we keep the old version around for > > > > a bit and offer the new version on top. Remember, that especially > > with non-compiletime bindings, we can not know which client is > > at what api version. > > > > _______________________________________________ > > hawkular-dev mailing list > > hawkular-dev(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hawkular-dev > > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

OK, so I think everyone has valid points and mostly I agree with Lukas, our issues are mainly the result of many quickly-iterating moving parts. But we do have room to improve. We need to remember that the reason we are running with all of these components is so that Hawkular master can be releasable both at any-time (stability) and at scheduled dates meeting certain feature goals (progress). With that in mind it's quite important for the [component] developers to have a clear picture or what is needed from them for the next Hawkular feature release. Given that knowledge the component should have a release available as far in advance as possible with the needed feature support. As Lukas mentions, 2 weeks is a very good goal. It could be that a Hawkular release doesn't even require a new component release. Releasing daily/weekly/etc really doesn't make much difference, I think, instead it is important that there is a release early enough in the Hawkular release cycle that supports the next release of Hawkular. This then sets us up to quickly respond with a follow-up release to address issues or things we missed. The components must strive to be out in front of the console. As for Mazz's suggestion of a dev/latest branch for each component. I'm not convinced that that is necessary. I think a component's master branch should be free to have commits towards their next release. And note, they should be ready to release a new feature release at any time. And that means no snapshot dependencies. So, if your component is currently dependent on a snapshot of another component, you will obviously need to be working in an integration branch. This is when Heiko's release-often approach is most critical. If someone is depending on your snapshot I think your highest priority should then be to release to get them off of that dependency. And consumers, help them do that as well. This is the coin-flip exercise. As for Versioning, we're getting closer to a standard but I think we could maybe do a little better. My take on the recent discussions is this. We know we can't release with snapshot dependencies so, as described above, we need to minimize the windows that a snapshot dep is in use by a consuming component. We are all using Major.Minor.Micro versioning. That is good. And most of us are following the RH guideline of the suffix, like ".Final". We are, I think, all using the following guidelines: * Breaking change releases minimally require a Minor version increment * Backward compatible minimally require a Micro version increment * Pure fix/patch releases should only increment the Micro version. I would further suggest that all releases at this time use the ".Final" suffix. Using ".AlphaX" does nothing more than create controversy. It has no meaning wrt quality or usability at this point in time. Anytime a feature or fix release goes out, the Major.Minor.Micro version change should be sufficient. Release often [enough], increment the version and tack on ".Final". No "Alpha", no "Snapshot", no timed "Snapshot". I think we should take to heart Heiko's intentions and release often. But I hope we can avoid fabricated release schedules for the components and instead have us focus on releasing based on the overall needs of Hawkular (2 weeks early!) and consuming components (kill snapshot deps ASAP!). On 7/21/2015 2:03 PM, Lukas Krejci wrote: > git-flow? > > On Tuesday, July 21, 2015 10:06:27 John Mazzitelli wrote: >> Each repo needs to have a "dev/latest" branch. Should the component need >> them, the pom.xml files in those dev/latest branches can refer to the >> latest-n-greatest snapshot versions of components (we can have travis >> publish the snapshot builds from these dev/latest branches). Therefore, if >> you want the latest-n-greatest to test integration, switch to dev/latest >> branches and build them locally. >> >> If you want a released version that never changes go to the master branch >> (which is the latest release) or go to one of the release tags and build >> from that. >> >> Once we are getting ready to prepare for a release, we create PRs to merge >> dev/latest to master, changing the poms to the release versions that are >> needed. >> >> How long between merging dev/latest to master and release? I dunno. If we do >> this right, we won't need a quick 1-week turnaround time. Every 2-weeks? >> 3-weeks? keep it a month? I don't know. Make something up. I vote every 17 >> days - only because its just as arbitrary as any other time period we might >> pick :) >> >> ----- Original Message ----- >> >>> Hello, >>> >>> There are two issues that get interlaced here and should not be. One is to >>> release individual components on a schedule, and second is making sure >>> that >>> all the components integrate nicely. Your proposal with "at least one >>> release per week" is trying to fix a fix of a fix of previous decision. >>> And >>> in this mix, the two concepts of release and integration get so >>> interlocked >>> that we cannot make heads and tail. >>> >>> Here is the progression of things: >>> >>> Problem 1: monolithic code is bad, we need to componentize everything. >>> Solution: create single purpose repositories, which resulted in about 20 >>> repositories. >>> >>> Problem 2: how do we integrate all those components? Solution: create a >>> Hawkular repository that depends on all the other repositories. >>> >>> Problem 3: integrated project idea works, but how do we really really >>> integrate the code? Solution: publish changes in subcomponents as soon as >>> possible. >>> >>> Problem 4: publishing changes in components takes a long time, can we >>> expedite the process and get changes even faster to integrated project? >>> Solution: automate SNAPSHOT publication so you have freshly build binaries >>> on almost every change. >>> >>> Problem 5: We need consistent builds for the integrated project. Solution: >>> master branch in the integrated project only depends on released version >>> of >>> components, feature branches are short lived. >>> >>> Problem 6: It is almost impossible to align all the release of components >>> and give enough runaway for the integrated project to ingest all the >>> release subcomponets, especially since there inter-dependencies on the >>> sub-projects themselves. Solution: publish sub-components officially at >>> least once per week. >>> >>> Problem 7: There are major changes that are needed right-away in the >>> integrated project. Solution: officially publish components more often >>> than >>> once per week. Publish as soon as the features makes it in. In fact, let's >>> publish Alphas (Alphaxx) with every single change. >>> >>> Problem 8: Too many releases, it takes too much time to administer the >>> process of releasing. Solution: Automate the release dependency >>> management, >>> emulate the SNAPSHOT injection concept but with Alpha "moniker". >>> >>> Problem 9: It is a nightmare to maintain on which version to depend on the >>> integrate project or components that depend on other components. Solution: >>> ?? >>> >>> Problem 10: It is almost impossible to trace back the code that code that >>> goes into the integrated project because there are so many Alphas. >>> Solution: ??? >>> >>> >>> We are at level 5; from 6 forward I see them coming, just as I saw the >>> rest >>> of 5. At what point do we stop and say "wait a second! what we doing here? >>> can we simplify all this??" From my perspective, we are arbitrarily >>> setting >>> requirements just to complicate things. There is always a happy medium and >>> there is always ways to simplify. I would much rather think for 2 months >>> on >>> how to simplify things and do it once, then stack problems that make our >>> lives harder for no reason. >>> >>> >>> That being said, I am not against implementing your proposed solution in >>> Hawkular Metrics. But I see two possible paths. One, escalate and resolve >>> problems 6, and 7 at the same time with automation. That is primarily >>> because we are so thin on resources and so stretched that we do not have >>> time to not automate this. Two, get a release engineer (or another >>> engineer) allocated to the team that will take care of these releases. If >>> we are to proceed with your current proposal please let us know which one >>> of these paths do you want Hawkular Metrics to take. >>> >>> >>> At the same time, we can take the reversed path; rather than escalate >>> problems, remove them from that stack. Why not look to simplify >>> everything, >>> revert a few problems and try to improve our productivity. We can revert >>> to >>> the use of SNAPSHOTS. In what capacity? Let's find the path that gives us >>> the most benefits with the least amount of work. >>> >>> >>> Thank you, >>> Stefan >>> >>> ----- Original Message ----- >>> >>>> From: "Heiko W.Rupp&quot;&lt;hrupp(a)redhat.com&gt; >>>> To: "Discussions around Hawkular development" >>>> <hawkular-dev(a)lists.jboss.org&gt; >>>> Sent: Tuesday, July 21, 2015 4:38:58 AM >>>> Subject: [Hawkular-dev] Release cadence >>>> >>>> Hey >>>> >>>> I have observed that our current Hawkular cadence of 4 weeks >>>> with similar cadences of components makes us end up with >>>> long living integration branches and a larger rush near the >>>> end to integrate them, get them for the first time tested in CI >>>> and even for the first time tested in real world. >>>> >>>> In one of the last releases there was a changed implementation >>>> in one component, that basically turned out as a no-op and >>>> still returned a "200 OK" code, so clients thought everything is >>>> happy, but it was not. We found the issue (through ppl looking >>>> at the UI) and solved it, but it was in a rush. >>>> >>>> This certainly goes against all the ideas of "release early, release >>>> often", "cut small slices", "changes go into CI/CD and go live quickly". >>>> Remember the coin flipping ? >>>> >>>> Ideally we would always be able to integrate changes from >>>> components into Hawkular (main), but I understand that with the >>>> way maven and its release process to central works, it is also not >>>> ideal to release many versions per day. >>>> >>>> With all of the above in mind, I propose that we move to a >>>> "at least once per week" model, where we do a component release >>>> at least once per week(*), which then in the four week stream form >>>> a new Alpha release. The smaller releases do not need release notes, >>>> I don't care if we use the micro number or a .AlphaY designator on >>>> them, but they should be a release, that is not a (named) snapshot. >>>> This will allow us to still have less efforts to do releases, but >>>> keep being (more) agile and have earlier integrations and thus >>>> less long living integration branches. >>>> >>>> On top of that, we need to provide new and/or changed apis(**) >>>> early on in the 4 weeks cadence so that other components can >>>> already start calling them, even if they are not yet functionally >>>> complete. >>>> >>>> *) Of course only if a change to the component has been made. >>>> **) Ideally with changed apis, we keep the old version around for >>>> >>>> a bit and offer the new version on top. Remember, that especially >>>> with non-compiletime bindings, we can not know which client is >>>> at what api version. >>>> >>>> _______________________________________________ >>>> hawkular-dev mailing list >>>> hawkular-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/hawkular-dev >>> _______________________________________________ >>> hawkular-dev mailing list >>> hawkular-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/hawkular-dev >> _______________________________________________ >> hawkular-dev mailing list >> hawkular-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/hawkular-dev > _______________________________________________ > hawkular-dev mailing list > hawkular-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hawkular-dev > _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev

Hey I have observed that our current Hawkular cadence of 4 weeks with similar cadences of components makes us end up with long living integration branches and a larger rush near the end to integrate them, get them for the first time tested in CI and even for the first time tested in real world. In one of the last releases there was a changed implementation in one component, that basically turned out as a no-op and still returned a "200 OK" code, so clients thought everything is happy, but it was not. We found the issue (through ppl looking at the UI) and solved it, but it was in a rush. This certainly goes against all the ideas of "release early, release often", "cut small slices", "changes go into CI/CD and go live quickly". Remember the coin flipping ? Ideally we would always be able to integrate changes from components into Hawkular (main), but I understand that with the way maven and its release process to central works, it is also not ideal to release many versions per day. With all of the above in mind, I propose that we move to a "at least once per week" model, where we do a component release at least once per week(*), which then in the four week stream form a new Alpha release. The smaller releases do not need release notes, I don't care if we use the micro number or a .AlphaY designator on them, but they should be a release, that is not a (named) snapshot. This will allow us to still have less efforts to do releases, but keep being (more) agile and have earlier integrations and thus less long living integration branches. On top of that, we need to provide new and/or changed apis(**) early on in the 4 weeks cadence so that other components can already start calling them, even if they are not yet functionally complete. *) Of course only if a change to the component has been made. **) Ideally with changed apis, we keep the old version around for a bit and offer the new version on top. Remember, that especially with non-compiletime bindings, we can not know which client is at what api version. _______________________________________________ hawkular-dev mailing list hawkular-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hawkular-dev