2:47 p.m.
Hi everyone!
In last few months we have been working on improving our pull request
testing & reporting.
As most of you know we have been running many test scenarios on "my"
teamcity instance at http://teamcity.cafe-babe-org and it has proven that
it works very well.
In many cases much better than what we had with jenkins on lightning.
That is why new solution is done on top of TeamCity rather than Jenkins.
Today we are turning off lightning's cron jobs for processing and turning
on new pull request building & github integration that we build on top of
TeamCity.
Given that reinstalation of the system(we also got some new disks that
require us to reconfigure RAID) is not trivial procedure from server uptime
point of view, we have setup pull processing to be done on "my" teamcity
while lightning will be offline.
As precaution we have just stopped pull player on lightning and we wont be
taking it offline until tomorrow evening (EST) given that new system works
as expected.
You can login to teamcity as guest, there is a little link on login page
and you don't need account created for watching build logs/progress.
So if you notice any problems in coming days let me know.
cheers,
Tomaz
8:33 a.m.
I'll let Tomaz answer that question, but I'll add some points where I
think TeamCity helped a lot recently when compared to Jenkins.
Just before 8.0.0.Alpha2 of WildFly was released, we noticed that our
testsuite was in a very bad shape. Too many intermittent failures.
Although, the intermittent failures weren't a new thing, the frequency
and the number had both grown to an extent where we felt that we had to
start looking into each of those tests and investigate the problems. I
hadn't used TeamCity earlier but while looking into these tests, I
decided to take a look at the instance maintained by Tomaz. From an user
point of view, I found the following features extremely handy and in
fact these features did help me with better investigating the failures
and also not losing interest in trying to track down those test failures:
1) "Investigations" feature - TeamCity has this feature called
"investigations" which allows you to mark a (failed) test or an entire
build for investigation. The investigation can be assigned to a specific
user. Investigations can be auto resolved (the next time the build/test
succeeds) or can be manually resolved after investigating that failure.
This feature allowed me to keep track of a bunch of failing tests and
monitor their resolution over time. This is one step between finding a
failing test and creating a JIRA, since this intermediate step allowed
me to spend some time on that test to really understand what needs to be
fixed/changed for that test to pass. Once I knew what was needed, I
could then either fix it or file the JIRA assigned to the relevant
component/person.This also was one way of saying that this specific test
failure is a "known issue which is being investigated on by person X".
This way someone else can spend their time on some other test failure
investigation.
Investigations also allow "notes" to be attached to them which allowed
me to make a note of what I have investigated so far and what might be
the issue.
2) Immediate report and logs of failed tests - Unlike Jenkins where you
have to wait for the entire testsuite to finish (which can take a hour
and a half) to know how many and which tests failed in that run,
TeamCity shows the progress of the build and reports the number of
failed tests at that point in time in the build. Furthermore, it shows
logs and the failure details of such tests immediately and you don't
have to wait for the run to complete. I found this extremely useful
since I didn't have to wait for the entire run to complete. In the past,
when I've seen intermittent failing tests on Jenkins, I haven't had the
determination to try out certain things and check the results since the
thought of having to wait for another hour and a half would just switch
off my interest on that issue.
3) Inline logs/stacktrace - I'm not sure why I like this so much but I
really do like this feature of TeamCity. This and #2 in themselves are
the reasons which kept me interested in tracking down a majority of the
failures. This specific feature is really simple. When a bunch of tests
fail in a build, the build report page shows all those failed test names
and also for each failed tests allows you to hide/show stacktrace inline
under the testcase name (see this for example
http://teamcity.cafe-babe.org/viewLog.html?buildId=5666&tab=buildResu...
- click on that test link and it will show up the logs inline and you
can then hide the logs if you want to). This allowed me to view all
those failed tests and their logs on the same page and hide whichever
ones I didn't want to view. Of course, in Jenkins, you can view these
logs on a separate page/tab for each failed test, but I find the
TeamCity way, much more usable than the Jenkins way.
-Jaikiran
On Tuesday 02 July 2013 11:50 PM, Vojtech Juranek wrote:
Hi,
> it works very well.
> In many cases much better than what we had with jenkins on lightning.
could you be more specific please? (not going to try to persuade you to stay
with Jenkins, just wondering what you see as Jenkins weak points and where is
TC better)
Thanks
Vojta
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
9:43 a.m.
One of the other things was that Jenkins uses stats that are relative to the last good
build, which makes them useless for determining how often an intermittent test fails. With
TC we immediately know what the top abusers are.
On Jul 3, 2013, at 8:33 AM, Jaikiran Pai <jpai(a)redhat.com> wrote:
I'll let Tomaz answer that question, but I'll add some points
where I think TeamCity helped a lot recently when compared to Jenkins.
Just before 8.0.0.Alpha2 of WildFly was released, we noticed that our testsuite was in a
very bad shape. Too many intermittent failures. Although, the intermittent failures
weren't a new thing, the frequency and the number had both grown to an extent where we
felt that we had to start looking into each of those tests and investigate the problems. I
hadn't used TeamCity earlier but while looking into these tests, I decided to take a
look at the instance maintained by Tomaz. From an user point of view, I found the
following features extremely handy and in fact these features did help me with better
investigating the failures and also not losing interest in trying to track down those test
failures:
1) "Investigations" feature - TeamCity has this feature called
"investigations" which allows you to mark a (failed) test or an entire build for
investigation. The investigation can be assigned to a specific user. Investigations can be
auto resolved (the next time the build/test succeeds) or can be manually resolved after
investigating that failure. This feature allowed me to keep track of a bunch of failing
tests and monitor their resolution over time. This is one step between finding a failing
test and creating a JIRA, since this intermediate step allowed me to spend some time on
that test to really understand what needs to be fixed/changed for that test to pass. Once
I knew what was needed, I could then either fix it or file the JIRA assigned to the
relevant component/person. This also was one way of saying that this specific test failure
is a "known issue which is being investigated on by person X". This way someone
else can spend their time on some other test failure investigation.
Investigations also allow "notes" to be attached to them which allowed me to
make a note of what I have investigated so far and what might be the issue.
2) Immediate report and logs of failed tests - Unlike Jenkins where you have to wait for
the entire testsuite to finish (which can take a hour and a half) to know how many and
which tests failed in that run, TeamCity shows the progress of the build and reports the
number of failed tests at that point in time in the build. Furthermore, it shows logs and
the failure details of such tests immediately and you don't have to wait for the run
to complete. I found this extremely useful since I didn't have to wait for the entire
run to complete. In the past, when I've seen intermittent failing tests on Jenkins, I
haven't had the determination to try out certain things and check the results since
the thought of having to wait for another hour and a half would just switch off my
interest on that issue.
3) Inline logs/stacktrace - I'm not sure why I like this so much but I really do like
this feature of TeamCity. This and #2 in themselves are the reasons which kept me
interested in tracking down a majority of the failures. This specific feature is really
simple. When a bunch of tests fail in a build, the build report page shows all those
failed test names and also for each failed tests allows you to hide/show stacktrace inline
under the testcase name (see this for example
http://teamcity.cafe-babe.org/viewLog.html?buildId=5666&tab=buildResu...
- click on that test link and it will show up the logs inline and you can then hide the
logs if you want to). This allowed me to view all those failed tests and their logs on the
same page and hide whichever ones I didn't want to view. Of course, in Jenkins, you
can view these logs on a separate page/tab for each failed test, but I find the TeamCity
way, much more usable than the Jenkins way.
-Jaikiran
On Tuesday 02 July 2013 11:50 PM, Vojtech Juranek wrote:
> Hi,
>
>
>> it works very well.
>> In many cases much better than what we had with jenkins on lightning.
>>
> could you be more specific please? (not going to try to persuade you to stay
> with Jenkins, just wondering what you see as Jenkins weak points and where is
> TC better)
>
> Thanks
> Vojta
>
>
>
>
> _______________________________________________
> wildfly-dev mailing list
>
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat
9:23 a.m.
Thanks for the feedback. While taking responsibility for failed test is IMHO
addressed by Claim plugin [1], the rest are useful proposals for better
Jenkins UX, now logged as [2,3,4]. If someone here is still interested in
Jenkins, feel free to log other proposals under JBQA-8165 [5].
Thanks
Vojta
[1] https://wiki.jenkins-ci.org/display/JENKINS/Claim+plugin
[2] https://issues.jboss.org/browse/JBQA-8167
[3] https://issues.jboss.org/browse/JBQA-8168
[4] https://issues.jboss.org/browse/JBQA-8169
[5] https://issues.jboss.org/browse/JBQA-8165
On Wednesday 03 July 2013 09:43:11 Jason Greene wrote:
One of the other things was that Jenkins uses stats that are relative
to the
last good build, which makes them useless for determining how often an
intermittent test fails. With TC we immediately know what the top abusers
are.
On Jul 3, 2013, at 8:33 AM, Jaikiran Pai <jpai(a)redhat.com> wrote:
> I'll let Tomaz answer that question, but I'll add some points where I
> think TeamCity helped a lot recently when compared to Jenkins.
>
> Just before 8.0.0.Alpha2 of WildFly was released, we noticed that our
> testsuite was in a very bad shape. Too many intermittent failures.
> Although, the intermittent failures weren't a new thing, the frequency
> and the number had both grown to an extent where we felt that we had to
> start looking into each of those tests and investigate the problems. I
> hadn't used TeamCity earlier but while looking into these tests, I
> decided to take a look at the instance maintained by Tomaz. From an user
> point of view, I found the following features extremely handy and in fact
> these features did help me with better investigating the failures and
> also not losing interest in trying to track down those test failures:
>
> 1) "Investigations" feature - TeamCity has this feature called
> "investigations" which allows you to mark a (failed) test or an entire
> build for investigation. The investigation can be assigned to a specific
> user. Investigations can be auto resolved (the next time the build/test
> succeeds) or can be manually resolved after investigating that failure.
> This feature allowed me to keep track of a bunch of failing tests and
> monitor their resolution over time. This is one step between finding a
> failing test and creating a JIRA, since this intermediate step allowed me
> to spend some time on that test to really understand what needs to be
> fixed/changed for that test to pass. Once I knew what was needed, I could
> then either fix it or file the JIRA assigned to the relevant
> component/person. This also was one way of saying that this specific test
> failure is a "known issue which is being investigated on by person X".
> This way someone else can spend their time on some other test failure
> investigation.
>
> Investigations also allow "notes" to be attached to them which allowed me
> to make a note of what I have investigated so far and what might be the
> issue.
>
> 2) Immediate report and logs of failed tests - Unlike Jenkins where you
> have to wait for the entire testsuite to finish (which can take a hour
> and a half) to know how many and which tests failed in that run, TeamCity
> shows the progress of the build and reports the number of failed tests at
> that point in time in the build. Furthermore, it shows logs and the
> failure details of such tests immediately and you don't have to wait for
> the run to complete. I found this extremely useful since I didn't have to
> wait for the entire run to complete. In the past, when I've seen
> intermittent failing tests on Jenkins, I haven't had the determination to
> try out certain things and check the results since the thought of having
> to wait for another hour and a half would just switch off my interest on
> that issue.
>
> 3) Inline logs/stacktrace - I'm not sure why I like this so much but I
> really do like this feature of TeamCity. This and #2 in themselves are
> the reasons which kept me interested in tracking down a majority of the
> failures. This specific feature is really simple. When a bunch of tests
> fail in a build, the build report page shows all those failed test names
> and also for each failed tests allows you to hide/show stacktrace inline
> under the testcase name (see this for example
> http://teamcity.cafe-babe.org/viewLog.html?buildId=5666&tab=buildResu...
> iv&buildTypeId=pr - click on that test link and it will show up the logs
> inline and you can then hide the logs if you want to). This allowed me to
> view all those failed tests and their logs on the same page and hide
> whichever ones I didn't want to view. Of course, in Jenkins, you can view
> these logs on a separate page/tab for each failed test, but I find the
> TeamCity way, much more usable than the Jenkins way.
>
> -Jaikiran
>
> On Tuesday 02 July 2013 11:50 PM, Vojtech Juranek wrote:
>> Hi,
>>
>>> it works very well.
>>> In many cases much better than what we had with jenkins on lightning.
>>
>> could you be more specific please? (not going to try to persuade you to
>> stay with Jenkins, just wondering what you see as Jenkins weak points
>> and where is TC better)
>>
>> Thanks
>> Vojta
>>
>>
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>>
>> wildfly-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat
9:33 a.m.
Thanks Vojta for filing those JIRAs. I just had a quick look at the
Claim plugin overview and it doesn't look like it allows the
investigation/claim to be done on a per testcase basis. Does it have
that support too?
-Jaikiran
On Monday 08 July 2013 07:53 PM, Vojtech Juranek wrote:
Thanks for the feedback. While taking responsibility for failed test
is IMHO
addressed by Claim plugin [1], the rest are useful proposals for better
Jenkins UX, now logged as [2,3,4]. If someone here is still interested in
Jenkins, feel free to log other proposals under JBQA-8165 [5].
Thanks
Vojta
[1] https://wiki.jenkins-ci.org/display/JENKINS/Claim+plugin
[2] https://issues.jboss.org/browse/JBQA-8167
[3] https://issues.jboss.org/browse/JBQA-8168
[4] https://issues.jboss.org/browse/JBQA-8169
[5] https://issues.jboss.org/browse/JBQA-8165
On Wednesday 03 July 2013 09:43:11 Jason Greene wrote:
> One of the other things was that Jenkins uses stats that are relative to the
> last good build, which makes them useless for determining how often an
> intermittent test fails. With TC we immediately know what the top abusers
> are.
> On Jul 3, 2013, at 8:33 AM, Jaikiran Pai <jpai(a)redhat.com> wrote:
>> I'll let Tomaz answer that question, but I'll add some points where I
>> think TeamCity helped a lot recently when compared to Jenkins.
>>
>> Just before 8.0.0.Alpha2 of WildFly was released, we noticed that our
>> testsuite was in a very bad shape. Too many intermittent failures.
>> Although, the intermittent failures weren't a new thing, the frequency
>> and the number had both grown to an extent where we felt that we had to
>> start looking into each of those tests and investigate the problems. I
>> hadn't used TeamCity earlier but while looking into these tests, I
>> decided to take a look at the instance maintained by Tomaz. From an user
>> point of view, I found the following features extremely handy and in fact
>> these features did help me with better investigating the failures and
>> also not losing interest in trying to track down those test failures:
>>
>> 1) "Investigations" feature - TeamCity has this feature called
>> "investigations" which allows you to mark a (failed) test or an entire
>> build for investigation. The investigation can be assigned to a specific
>> user. Investigations can be auto resolved (the next time the build/test
>> succeeds) or can be manually resolved after investigating that failure.
>> This feature allowed me to keep track of a bunch of failing tests and
>> monitor their resolution over time. This is one step between finding a
>> failing test and creating a JIRA, since this intermediate step allowed me
>> to spend some time on that test to really understand what needs to be
>> fixed/changed for that test to pass. Once I knew what was needed, I could
>> then either fix it or file the JIRA assigned to the relevant
>> component/person. This also was one way of saying that this specific test
>> failure is a "known issue which is being investigated on by person X".
>> This way someone else can spend their time on some other test failure
>> investigation.
>>
>> Investigations also allow "notes" to be attached to them which allowed
me
>> to make a note of what I have investigated so far and what might be the
>> issue.
>>
>> 2) Immediate report and logs of failed tests - Unlike Jenkins where you
>> have to wait for the entire testsuite to finish (which can take a hour
>> and a half) to know how many and which tests failed in that run, TeamCity
>> shows the progress of the build and reports the number of failed tests at
>> that point in time in the build. Furthermore, it shows logs and the
>> failure details of such tests immediately and you don't have to wait for
>> the run to complete. I found this extremely useful since I didn't have to
>> wait for the entire run to complete. In the past, when I've seen
>> intermittent failing tests on Jenkins, I haven't had the determination to
>> try out certain things and check the results since the thought of having
>> to wait for another hour and a half would just switch off my interest on
>> that issue.
>>
>> 3) Inline logs/stacktrace - I'm not sure why I like this so much but I
>> really do like this feature of TeamCity. This and #2 in themselves are
>> the reasons which kept me interested in tracking down a majority of the
>> failures. This specific feature is really simple. When a bunch of tests
>> fail in a build, the build report page shows all those failed test names
>> and also for each failed tests allows you to hide/show stacktrace inline
>> under the testcase name (see this for example
>> http://teamcity.cafe-babe.org/viewLog.html?buildId=5666&tab=buildResu...
>> iv&buildTypeId=pr - click on that test link and it will show up the logs
>> inline and you can then hide the logs if you want to). This allowed me to
>> view all those failed tests and their logs on the same page and hide
>> whichever ones I didn't want to view. Of course, in Jenkins, you can view
>> these logs on a separate page/tab for each failed test, but I find the
>> TeamCity way, much more usable than the Jenkins way.
>>
>> -Jaikiran
>>
>> On Tuesday 02 July 2013 11:50 PM, Vojtech Juranek wrote:
>>> Hi,
>>>
>>>> it works very well.
>>>> In many cases much better than what we had with jenkins on lightning.
>>> could you be more specific please? (not going to try to persuade you to
>>> stay with Jenkins, just wondering what you see as Jenkins weak points
>>> and where is TC better)
>>>
>>> Thanks
>>> Vojta
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> wildfly-dev mailing list
>>>
>>> wildfly-dev(a)lists.jboss.org
>>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
> --
> Jason T. Greene
> WildFly Lead / JBoss EAP Platform Architect
> JBoss, a division of Red Hat
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
11:24 a.m.
On Monday 08 July 2013 20:03:38 Jaikiran Pai wrote:
Claim plugin overview and it doesn't look like it allows the
investigation/claim to be done on a per testcase basis. Does it have
that support too?
you are right, it doesn't support assignment per testcast (I don't use it, so
I also tried now)
9:54 a.m.
Hi,
to add few other observations I had with Jenkins vs TeamCity.
Biggest issue from my point of view is that Jenkins has a big performance
issue when running more than one job same in concurrently on different
nodes.
What happens is that when you have lets say 5 same jobs (with different
parameters) running and first job would get to the end it would get
stuck/wait in "recording test results" until all jobs are done and then all
of them would finish in matter of minutes.
That causes big issue as build slaves don't do noting for in some cases
hours as they are waiting for other jobs to complete. And new jobs are
waiting for avalibale slaves, this result in hardware not being utilized as
much as it could be.
Related recorded issues for this
https://issues.jenkins-ci.org/browse/JENKINS-10234
https://issues.jenkins-ci.org/browse/JENKINS-10175
Issues are open for few years now and the answer to it was "this is a
feature not a bug"
everyone is recommending using xunit instead of junit plugin, but trick is
that xunit plugin has totally different semantics and as such does not
resolve our issue.
Other problem is general performance of building and amount it loads
hardware. In general it takes 10-15% more time to do same job on jenkins
then on TC when running on same hardware.
This are just few deep technical issues I have found.
In any case i think everyone agrees that UI in general is much better with
TC and also TC has nice integration with IDEs (eclipse, intellij, VS.NET)
which allow you to monitor/debug/start/review builds as they are executing.
That add a nice feature for developers to debug why their code is failing
just one one build slave for example.
--
tomaz
On Wed, Jul 3, 2013 at 3:33 PM, Jaikiran Pai <jpai(a)redhat.com> wrote:
I'll let Tomaz answer that question, but I'll add some
points where I
think TeamCity helped a lot recently when compared to Jenkins.
Just before 8.0.0.Alpha2 of WildFly was released, we noticed that our
testsuite was in a very bad shape. Too many intermittent failures.
Although, the intermittent failures weren't a new thing, the frequency and
the number had both grown to an extent where we felt that we had to start
looking into each of those tests and investigate the problems. I hadn't
used TeamCity earlier but while looking into these tests, I decided to take
a look at the instance maintained by Tomaz. From an user point of view, I
found the following features extremely handy and in fact these features did
help me with better investigating the failures and also not losing interest
in trying to track down those test failures:
1) "Investigations" feature - TeamCity has this feature called
"investigations" which allows you to mark a (failed) test or an entire
build for investigation. The investigation can be assigned to a specific
user. Investigations can be auto resolved (the next time the build/test
succeeds) or can be manually resolved after investigating that failure.
This feature allowed me to keep track of a bunch of failing tests and
monitor their resolution over time. This is one step between finding a
failing test and creating a JIRA, since this intermediate step allowed me
to spend some time on that test to really understand what needs to be
fixed/changed for that test to pass. Once I knew what was needed, I could
then either fix it or file the JIRA assigned to the relevant
component/person. This also was one way of saying that this specific test
failure is a "known issue which is being investigated on by person X". This
way someone else can spend their time on some other test failure
investigation.
Investigations also allow "notes" to be attached to them which allowed
me to make a note of what I have investigated so far and what might be the
issue.
2) Immediate report and logs of failed tests - Unlike Jenkins where you
have to wait for the entire testsuite to finish (which can take a hour and
a half) to know how many and which tests failed in that run, TeamCity shows
the progress of the build and reports the number of failed tests at that
point in time in the build. Furthermore, it shows logs and the failure
details of such tests immediately and you don't have to wait for the run to
complete. I found this extremely useful since I didn't have to wait for the
entire run to complete. In the past, when I've seen intermittent failing
tests on Jenkins, I haven't had the determination to try out certain things
and check the results since the thought of having to wait for another hour
and a half would just switch off my interest on that issue.
3) Inline logs/stacktrace - I'm not sure why I like this so much but I
really do like this feature of TeamCity. This and #2 in themselves are the
reasons which kept me interested in tracking down a majority of the
failures. This specific feature is really simple. When a bunch of tests
fail in a build, the build report page shows all those failed test names
and also for each failed tests allows you to hide/show stacktrace inline
under the testcase name (see this for example
http://teamcity.cafe-babe.org/viewLog.html?buildId=5666&tab=buildResu...
click on that test link and it will show up the logs inline and you can
then hide the logs if you want to). This allowed me to view all those
failed tests and their logs on the same page and hide whichever ones I
didn't want to view. Of course, in Jenkins, you can view these logs on a
separate page/tab for each failed test, but I find the TeamCity way, much
more usable than the Jenkins way.
-Jaikiran
On Tuesday 02 July 2013 11:50 PM, Vojtech Juranek wrote:
Hi,
it works very well.
In many cases much better than what we had with jenkins on lightning.
could you be more specific please? (not going to try to persuade you to stay
with Jenkins, just wondering what you see as Jenkins weak points and where is
TC better)
Thanks
Vojta
_______________________________________________
wildfly-dev mailing
listwildfly-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
10 a.m.
On Monday 08 July 2013 08:24 PM, Tomaž Cerar wrote:
...
In any case i think everyone agrees that UI in general is much better
with TC
I found it to be the other way around. In fact, for me the UI for
Jenkins is almost non-existent whereas for TeamCity I did find it to be
good enough and intuitive for most parts of it.
-Jaikiran
--
tomaz
On Wed, Jul 3, 2013 at 3:33 PM, Jaikiran Pai <jpai(a)redhat.com
<mailto:jpai@redhat.com>>wrote:
I'll let Tomaz answer that question, but I'll add some points
where I think TeamCity helped a lot recently when compared to
Jenkins.
Just before 8.0.0.Alpha2 of WildFly was released, we noticed that
our testsuite was in a very bad shape. Too many intermittent
failures. Although, the intermittent failures weren't a new thing,
the frequency and the number had both grown to an extent where we
felt that we had to start looking into each of those tests and
investigate the problems. I hadn't used TeamCity earlier but while
looking into these tests, I decided to take a look at the instance
maintained by Tomaz. From an user point of view, I found the
following features extremely handy and in fact these features did
help me with better investigating the failures and also not losing
interest in trying to track down those test failures:
1) "Investigations" feature - TeamCity has this feature called
"investigations" which allows you to mark a (failed) test or an
entire build for investigation. The investigation can be assigned
to a specific user. Investigations can be auto resolved (the next
time the build/test succeeds) or can be manually resolved after
investigating that failure. This feature allowed me to keep track
of a bunch of failing tests and monitor their resolution over
time. This is one step between finding a failing test and creating
a JIRA, since this intermediate step allowed me to spend some time
on that test to really understand what needs to be fixed/changed
for that test to pass. Once I knew what was needed, I could then
either fix it or file the JIRA assigned to the relevant
component/person.This also was one way of saying that this
specific test failure is a "known issue which is being
investigated on by person X". This way someone else can spend
their time on some other test failure investigation.
Investigations also allow "notes" to be attached to them which
allowed me to make a note of what I have investigated so far and
what might be the issue.
2) Immediate report and logs of failed tests - Unlike Jenkins
where you have to wait for the entire testsuite to finish (which
can take a hour and a half) to know how many and which tests
failed in that run, TeamCity shows the progress of the build and
reports the number of failed tests at that point in time in the
build. Furthermore, it shows logs and the failure details of such
tests immediately and you don't have to wait for the run to
complete. I found this extremely useful since I didn't have to
wait for the entire run to complete. In the past, when I've seen
intermittent failing tests on Jenkins, I haven't had the
determination to try out certain things and check the results
since the thought of having to wait for another hour and a half
would just switch off my interest on that issue.
3) Inline logs/stacktrace - I'm not sure why I like this so much
but I really do like this feature of TeamCity. This and #2 in
themselves are the reasons which kept me interested in tracking
down a majority of the failures. This specific feature is really
simple. When a bunch of tests fail in a build, the build report
page shows all those failed test names and also for each failed
tests allows you to hide/show stacktrace inline under the testcase
name (see this for example
http://teamcity.cafe-babe.org/viewLog.html?buildId=5666&tab=buildResu...
click on that test link and it will show up the logs inline and
you can then hide the logs if you want to). This allowed me to
view all those failed tests and their logs on the same page and
hide whichever ones I didn't want to view. Of course, in Jenkins,
you can view these logs on a separate page/tab for each failed
test, but I find the TeamCity way, much more usable than the
Jenkins way.
-Jaikiran
On Tuesday 02 July 2013 11:50 PM, Vojtech Juranek wrote:
> Hi,
>
>> it works very well.
>> In many cases much better than what we had with jenkins on lightning.
> could you be more specific please? (not going to try to persuade you to stay
> with Jenkins, just wondering what you see as Jenkins weak points and where is
> TC better)
>
> Thanks
> Vojta
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
https://lists.jboss.org/mailman/listinfo/wildfly-dev
10:07 a.m.
On Monday 08 July 2013 08:30 PM, Jaikiran Pai wrote:
On Monday 08 July 2013 08:24 PM, Tomaž Cerar wrote:
>
> ...
> In any case i think everyone agrees that UI in general is much better
> with TC
I found it to be the other way around. In fact, for me the UI for
Jenkins is almost non-existent whereas for TeamCity I did find it to
be good enough and intuitive for most parts of it.
Never mind, I read your message wrong. We both are actually saying the
same thing - that TC UI is better :)
-Jaikiran
-Jaikiran
>
>
> --
> tomaz
>
>
>
>
> On Wed, Jul 3, 2013 at 3:33 PM, Jaikiran Pai <jpai(a)redhat.com
> <mailto:jpai@redhat.com>>wrote:
>
> I'll let Tomaz answer that question, but I'll add some points
> where I think TeamCity helped a lot recently when compared to
> Jenkins.
>
> Just before 8.0.0.Alpha2 of WildFly was released, we noticed that
> our testsuite was in a very bad shape. Too many intermittent
> failures. Although, the intermittent failures weren't a new
> thing, the frequency and the number had both grown to an extent
> where we felt that we had to start looking into each of those
> tests and investigate the problems. I hadn't used TeamCity
> earlier but while looking into these tests, I decided to take a
> look at the instance maintained by Tomaz. From an user point of
> view, I found the following features extremely handy and in fact
> these features did help me with better investigating the failures
> and also not losing interest in trying to track down those test
> failures:
>
> 1) "Investigations" feature - TeamCity has this feature called
> "investigations" which allows you to mark a (failed) test or an
> entire build for investigation. The investigation can be assigned
> to a specific user. Investigations can be auto resolved (the next
> time the build/test succeeds) or can be manually resolved after
> investigating that failure. This feature allowed me to keep track
> of a bunch of failing tests and monitor their resolution over
> time. This is one step between finding a failing test and
> creating a JIRA, since this intermediate step allowed me to spend
> some time on that test to really understand what needs to be
> fixed/changed for that test to pass. Once I knew what was needed,
> I could then either fix it or file the JIRA assigned to the
> relevant component/person.This also was one way of saying that
> this specific test failure is a "known issue which is being
> investigated on by person X". This way someone else can spend
> their time on some other test failure investigation.
>
> Investigations also allow "notes" to be attached to them which
> allowed me to make a note of what I have investigated so far and
> what might be the issue.
>
> 2) Immediate report and logs of failed tests - Unlike Jenkins
> where you have to wait for the entire testsuite to finish (which
> can take a hour and a half) to know how many and which tests
> failed in that run, TeamCity shows the progress of the build and
> reports the number of failed tests at that point in time in the
> build. Furthermore, it shows logs and the failure details of such
> tests immediately and you don't have to wait for the run to
> complete. I found this extremely useful since I didn't have to
> wait for the entire run to complete. In the past, when I've seen
> intermittent failing tests on Jenkins, I haven't had the
> determination to try out certain things and check the results
> since the thought of having to wait for another hour and a half
> would just switch off my interest on that issue.
>
> 3) Inline logs/stacktrace - I'm not sure why I like this so much
> but I really do like this feature of TeamCity. This and #2 in
> themselves are the reasons which kept me interested in tracking
> down a majority of the failures. This specific feature is really
> simple. When a bunch of tests fail in a build, the build report
> page shows all those failed test names and also for each failed
> tests allows you to hide/show stacktrace inline under the
> testcase name (see this for example
>
http://teamcity.cafe-babe.org/viewLog.html?buildId=5666&tab=buildResu...
> click on that test link and it will show up the logs inline and
> you can then hide the logs if you want to). This allowed me to
> view all those failed tests and their logs on the same page and
> hide whichever ones I didn't want to view. Of course, in Jenkins,
> you can view these logs on a separate page/tab for each failed
> test, but I find the TeamCity way, much more usable than the
> Jenkins way.
>
> -Jaikiran
>
>
> On Tuesday 02 July 2013 11:50 PM, Vojtech Juranek wrote:
>> Hi,
>>
>>> it works very well.
>>> In many cases much better than what we had with jenkins on lightning.
>> could you be more specific please? (not going to try to persuade you to stay
>> with Jenkins, just wondering what you see as Jenkins weak points and where
is
>> TC better)
>>
>> Thanks
>> Vojta
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
11:27 a.m.
On Jul 8, 2013, at 10:07 AM, Jaikiran Pai <jpai(a)redhat.com> wrote:
On Monday 08 July 2013 08:30 PM, Jaikiran Pai wrote:
> On Monday 08 July 2013 08:24 PM, Tomaž Cerar wrote:
>>
>> ...
>> In any case i think everyone agrees that UI in general is much better with TC
>
> I found it to be the other way around. In fact, for me the UI for Jenkins is almost
non-existent whereas for TeamCity I did find it to be good enough and intuitive for most
parts of it.
Never mind, I read your message wrong. We both are actually saying the same thing - that
TC UI is better :)
In fairness I was quite found of the status layout in Jenkins. Like a combination of the
Agent and test result views on the same screen
--
Jason T. Greene
WildFly Lead / JBoss EAP Platform Architect
JBoss, a division of Red Hat
10:22 a.m.
On 7/8/13 10:00 AM, Jaikiran Pai wrote:
On Monday 08 July 2013 08:24 PM, Tomaž Cerar wrote:
>
> ...
> In any case i think everyone agrees that UI in general is much better
> with TC
I found it to be the other way around. In fact, for me the UI for
Jenkins is almost non-existent whereas for TeamCity I did find it to be
good enough and intuitive for most parts of it.
Agreed, but one warning for new users of TC -- look for small, lightly
colored controls like a pull down arrow or an ellipsis. Less common
stuff you might wish was doable from a particular spot in the UI often
is available from one of these, but they seem to keep the UI uncluttered
looking by making these controls quite subtle.
-Jaikiran
>
>
> --
> tomaz
>
>
>
>
> On Wed, Jul 3, 2013 at 3:33 PM, Jaikiran Pai <jpai(a)redhat.com
> <mailto:jpai@redhat.com>>wrote:
>
> I'll let Tomaz answer that question, but I'll add some points
> where I think TeamCity helped a lot recently when compared to
> Jenkins.
>
> Just before 8.0.0.Alpha2 of WildFly was released, we noticed that
> our testsuite was in a very bad shape. Too many intermittent
> failures. Although, the intermittent failures weren't a new thing,
> the frequency and the number had both grown to an extent where we
> felt that we had to start looking into each of those tests and
> investigate the problems. I hadn't used TeamCity earlier but while
> looking into these tests, I decided to take a look at the instance
> maintained by Tomaz. From an user point of view, I found the
> following features extremely handy and in fact these features did
> help me with better investigating the failures and also not losing
> interest in trying to track down those test failures:
>
> 1) "Investigations" feature - TeamCity has this feature called
> "investigations" which allows you to mark a (failed) test or an
> entire build for investigation. The investigation can be assigned
> to a specific user. Investigations can be auto resolved (the next
> time the build/test succeeds) or can be manually resolved after
> investigating that failure. This feature allowed me to keep track
> of a bunch of failing tests and monitor their resolution over
> time. This is one step between finding a failing test and creating
> a JIRA, since this intermediate step allowed me to spend some time
> on that test to really understand what needs to be fixed/changed
> for that test to pass. Once I knew what was needed, I could then
> either fix it or file the JIRA assigned to the relevant
> component/person.This also was one way of saying that this
> specific test failure is a "known issue which is being
> investigated on by person X". This way someone else can spend
> their time on some other test failure investigation.
>
> Investigations also allow "notes" to be attached to them which
> allowed me to make a note of what I have investigated so far and
> what might be the issue.
>
> 2) Immediate report and logs of failed tests - Unlike Jenkins
> where you have to wait for the entire testsuite to finish (which
> can take a hour and a half) to know how many and which tests
> failed in that run, TeamCity shows the progress of the build and
> reports the number of failed tests at that point in time in the
> build. Furthermore, it shows logs and the failure details of such
> tests immediately and you don't have to wait for the run to
> complete. I found this extremely useful since I didn't have to
> wait for the entire run to complete. In the past, when I've seen
> intermittent failing tests on Jenkins, I haven't had the
> determination to try out certain things and check the results
> since the thought of having to wait for another hour and a half
> would just switch off my interest on that issue.
>
> 3) Inline logs/stacktrace - I'm not sure why I like this so much
> but I really do like this feature of TeamCity. This and #2 in
> themselves are the reasons which kept me interested in tracking
> down a majority of the failures. This specific feature is really
> simple. When a bunch of tests fail in a build, the build report
> page shows all those failed test names and also for each failed
> tests allows you to hide/show stacktrace inline under the testcase
> name (see this for example
>
http://teamcity.cafe-babe.org/viewLog.html?buildId=5666&tab=buildResu...
> click on that test link and it will show up the logs inline and
> you can then hide the logs if you want to). This allowed me to
> view all those failed tests and their logs on the same page and
> hide whichever ones I didn't want to view. Of course, in Jenkins,
> you can view these logs on a separate page/tab for each failed
> test, but I find the TeamCity way, much more usable than the
> Jenkins way.
>
> -Jaikiran
>
>
> On Tuesday 02 July 2013 11:50 PM, Vojtech Juranek wrote:
>> Hi,
>>
>>> it works very well.
>>> In many cases much better than what we had with jenkins on lightning.
>> could you be more specific please? (not going to try to persuade you to stay
>> with Jenkins, just wondering what you see as Jenkins weak points and where
is
>> TC better)
>>
>> Thanks
>> Vojta
>>
>>
>> _______________________________________________
>> wildfly-dev mailing list
>> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
>> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
> _______________________________________________
> wildfly-dev mailing list
> wildfly-dev(a)lists.jboss.org <mailto:wildfly-dev@lists.jboss.org>
> https://lists.jboss.org/mailman/listinfo/wildfly-dev
>
>
_______________________________________________
wildfly-dev mailing list
wildfly-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/wildfly-dev
--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
11:38 a.m.
yes, this one is IMHO valid issue, will look on it when time allows - make
regression analysis optional (and thus make synchronization of results
collection optional) seems doable. Or skip the analysis and do it on demand.
Other problem is general performance of building and amount it loads
hardware. In general it takes 10-15% more time to do same job on jenkins
then on TC when running on same hardware.
we had a discussion about it in the past and IIRC it happened _only_ on
windows and there was a time difference in running single tests, i.e. things
which have nothing to do with Jenkins, IMHO some environment misconfiguration
or something like this, not a Jenkins performance issue
3945
days inactive
3952
days old
12 comments
5 participants
participants (5)
-
Brian Stansberry -
Jaikiran Pai -
Jason Greene -
Tomaž Cerar -
Vojtech Juranek