8:58 a.m.
Hi all,
This is a warning since we plan a code freeze for tomorrow: a lot of
jobs are failing (timeout, or just compilation or assembling issue, or
Jenkins configuration). As developers on a component, you must react to
a build becoming red, so don't ignore Jenkins mail notifications, read
the log when a build fails and*fix it ASAP*, if you can. You may be
interested in receiving mails specifically for your component, so you
can add you mail address on the Configuration page, "Mail notifications"
section.
In some cases, log is explicit and shows what file crashed the build. So
you can guess what you need to do to fix it.
In other cases, it is be an environment issue, then open a ticket to the
Build/Releng component and assign it to Nick or I depending on your
timezone, so we can help to get the job back to work.
Since we are moving to bigger components, with more isolation, you
really need to be careful about the status of your component CI job.
Please don't wait for QE or Build crew to discover issues and fix it, it
simply won't scale.
Cheers,
--
Mickael Istria
Eclipse developer at JBoss, by Red Hat <http://www.jboss.org/tools>
My blog <http://mickaelistria.wordpress.com> - My Tweets
<http://twitter.com/mickaelistria>
7:51 p.m.
To be honest, I ignore Jenkins emails. My experience shows me that 95%
of these problems are problems of hudson build(s)/environment/... or
problems which I'm already aware of.
Instead of rely on the hudson builds I regularly run tycho build
locally. It safes our time. So instead of investigating what is going on
with hudson builds every time I got Jenkis email we (CDI/Seam/JSF team)
can spend time on development running local tycho build every other day
to make sure everything is ok on our side.
We mentioned these problems in JIRA and in emails but the hudson builds
is still unstable.
On 08/29/2012 11:58 PM, Mickael Istria wrote:
Hi all,
This is a warning since we plan a code freeze for tomorrow: a lot of
jobs are failing (timeout, or just compilation or assembling issue, or
Jenkins configuration). As developers on a component, you must react
to a build becoming red, so don't ignore Jenkins mail notifications,
read the log when a build fails and*fix it ASAP*, if you can. You may
be interested in receiving mails specifically for your component, so
you can add you mail address on the Configuration page, "Mail
notifications" section.
In some cases, log is explicit and shows what file crashed the build.
So you can guess what you need to do to fix it.
In other cases, it is be an environment issue, then open a ticket to
the Build/Releng component and assign it to Nick or I depending on
your timezone, so we can help to get the job back to work.
Since we are moving to bigger components, with more isolation, you
really need to be careful about the status of your component CI job.
Please don't wait for QE or Build crew to discover issues and fix it,
it simply won't scale.
Cheers,
--
Mickael Istria
Eclipse developer at JBoss, by Red Hat <http://www.jboss.org/tools>
My blog <http://mickaelistria.wordpress.com> - My Tweets
<http://twitter.com/mickaelistria>
_______________________________________________
jbosstools-dev mailing list
jbosstools-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jbosstools-dev
8:41 p.m.
Going forward I'm looking to have a better sequence of builds in
Jenkins. This is difficult when the whole stack can take more than 16hrs
to build, so having SVN checks every 6 hrs (as we do today) might not be
ideal if there's a ton of upstream changes coming down the pipe.
If you spread the 25 components out in approximate order, assuming that
no two run in parallel (which is of course not the case but makes it
easier to estimate times) then a fresh CDI build is over 9 hours after a
fresh usage, tests, or common build. Since CDI has over an hour of tests
in it, Seam is therefore close to 12 hours down the stack.
Thus if a change comes in to Common at 00:00, Seam might start building
at 12:00. If a change comes in to Common at 9:30 while Seam is spinning,
it won't spin again for another 12 hours, during which time CDI might
respin.
Now, if there were API changes in Common which affected both CDI and
Seam, only CDI will have seen the change (as Seam will still be pending)
and we could end up with a mismatch in the JBT aggregate. Or, as is
often the case you describe, you see the change in SVN, respond to it,
and have a fix in SVN HOURS before a build appears that picks up the
change. In between, you might get a compilation error email complaining
about the older version of the source.
One thing I was prototyping (only implemented partially) for the JBDS 5
builds was to use the Parameterized Build Trigger plugin instead of the
default Build Trigger plugin; this new one allows the same SVN revision
used upstream (eg., by Common) to be used downstream (eg., by Seam) so
that in theory the source versions would be compatible.
On further reflection, it occurs to me that this too could cause
problems because if a change is needed in Seam to react to a change in
Common, then the version of source used by Seam would be GREATER than
the version of the source used by Common; forcing them to be the same
would mean we'd be building against old sources. This would be another
reason why we'd get Jenkins emails complaining about test failures or
compiler errors which would not be seen when building/testing locally w/
Tycho.
On 08/30/2012 01:51 PM, Alexey Kazakov wrote:
To be honest, I ignore Jenkins emails. My experience shows me that
95%
of these problems are problems of hudson build(s)/environment/... or
problems which I'm already aware of.
Instead of rely on the hudson builds I regularly run tycho build
locally. It safes our time. So instead of investigating what is going on
with hudson builds every time I got Jenkis email we (CDI/Seam/JSF team)
can spend time on development running local tycho build every other day
to make sure everything is ok on our side.
We mentioned these problems in JIRA and in emails but the hudson builds
is still unstable.
On 08/29/2012 11:58 PM, Mickael Istria wrote:
> Hi all,
>
> This is a warning since we plan a code freeze for tomorrow: a lot of
> jobs are failing (timeout, or just compilation or assembling issue, or
> Jenkins configuration). As developers on a component, you must react
> to a build becoming red, so don't ignore Jenkins mail notifications,
> read the log when a build fails and*fix it ASAP*, if you can. You may
> be interested in receiving mails specifically for your component, so
> you can add you mail address on the Configuration page, "Mail
> notifications" section.
> In some cases, log is explicit and shows what file crashed the build.
> So you can guess what you need to do to fix it.
> In other cases, it is be an environment issue, then open a ticket to
> the Build/Releng component and assign it to Nick or I depending on
> your timezone, so we can help to get the job back to work.
>
> Since we are moving to bigger components, with more isolation, you
> really need to be careful about the status of your component CI job.
> Please don't wait for QE or Build crew to discover issues and fix it,
> it simply won't scale.
>
> Cheers,
> --
> Mickael Istria
> Eclipse developer at JBoss, by Red Hat <http://www.jboss.org/tools>
> My blog <http://mickaelistria.wordpress.com> - My Tweets
> <http://twitter.com/mickaelistria>
>
>
> _______________________________________________
> jbosstools-dev mailing list
> jbosstools-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jbosstools-dev
_______________________________________________
jbosstools-dev mailing list
jbosstools-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jbosstools-dev
--
Nick Boldt :: JBoss by Red Hat
Productization Lead :: JBoss Tools & Dev Studio
http://nick.divbyzero.com
9:02 p.m.
On 08/30/2012 11:41 AM, Nick Boldt wrote:
If you spread the 25 components out in approximate order, assuming
that no two run in parallel (which is of course not the case but makes
it easier to estimate times) then a fresh CDI build is over 9 hours
after a fresh usage, tests, or common build. Since CDI has over an
hour of tests in it, Seam is therefore close to 12 hours down the stack.
An hour for CDI tests? Why so long? It takes ~ 11 minutes (about 1000
JUnit tests) with Tycho on my laptop.
10:04 p.m.
Not sure... you tell me:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstools-3.3_stabl...
(3.3.x branch)
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstools-3.3_trunk...
(trunk)
On 08/30/2012 03:02 PM, Alexey Kazakov wrote:
On 08/30/2012 11:41 AM, Nick Boldt wrote:
> If you spread the 25 components out in approximate order, assuming
> that no two run in parallel (which is of course not the case but makes
> it easier to estimate times) then a fresh CDI build is over 9 hours
> after a fresh usage, tests, or common build. Since CDI has over an
> hour of tests in it, Seam is therefore close to 12 hours down the stack.
An hour for CDI tests? Why so long? It takes ~ 11 minutes (about 1000
JUnit tests) with Tycho on my laptop.
--
Nick Boldt :: JBoss by Red Hat
Productization Lead :: JBoss Tools & Dev Studio
http://nick.divbyzero.com
11:28 p.m.
That's what I found. The problem is in vncserver. Latest vnc in
combination with gnome manager makes tests waiting forever in UI thread
when importing projects and use JobMtils.waitForIdle(). It turned out to
make the tests move ahead I needed to open vnc display in viewer and
sent any event from keyboard of from mouse.
the fix was replacing in ~/.vnc/xstartup
gnome-session &
to
metacity --replace &
and tests are fine again.
I'm not sure but that might be the case.
Denis
On 08/30/2012 01:04 PM, Nick Boldt wrote:
Not sure... you tell me:
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstools-3.3_stabl...
(3.3.x branch)
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstools-3.3_trunk...
(trunk)
On 08/30/2012 03:02 PM, Alexey Kazakov wrote:
> On 08/30/2012 11:41 AM, Nick Boldt wrote:
>> If you spread the 25 components out in approximate order, assuming
>> that no two run in parallel (which is of course not the case but makes
>> it easier to estimate times) then a fresh CDI build is over 9 hours
>> after a fresh usage, tests, or common build. Since CDI has over an
>> hour of tests in it, Seam is therefore close to 12 hours down the stack.
> An hour for CDI tests? Why so long? It takes ~ 11 minutes (about 1000
> JUnit tests) with Tycho on my laptop.
9:10 a.m.
have this change been done on our jenkins instance ?
/max
On 30 Aug 2012, at 23:28, Denis Golovin <dgolovin(a)exadel.com> wrote:
That's what I found. The problem is in vncserver. Latest vnc in
combination with gnome manager makes tests waiting forever in UI thread
when importing projects and use JobMtils.waitForIdle(). It turned out to
make the tests move ahead I needed to open vnc display in viewer and
sent any event from keyboard of from mouse.
the fix was replacing in ~/.vnc/xstartup
gnome-session &
to
metacity --replace &
and tests are fine again.
I'm not sure but that might be the case.
Denis
On 08/30/2012 01:04 PM, Nick Boldt wrote:
> Not sure... you tell me:
>
>
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstools-3.3_stabl...
> (3.3.x branch)
>
>
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstools-3.3_trunk...
> (trunk)
>
>
> On 08/30/2012 03:02 PM, Alexey Kazakov wrote:
>> On 08/30/2012 11:41 AM, Nick Boldt wrote:
>>> If you spread the 25 components out in approximate order, assuming
>>> that no two run in parallel (which is of course not the case but makes
>>> it easier to estimate times) then a fresh CDI build is over 9 hours
>>> after a fresh usage, tests, or common build. Since CDI has over an
>>> hour of tests in it, Seam is therefore close to 12 hours down the stack.
>> An hour for CDI tests? Why so long? It takes ~ 11 minutes (about 1000
>> JUnit tests) with Tycho on my laptop.
_______________________________________________
jbosstools-dev mailing list
jbosstools-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jbosstools-dev
10:46 a.m.
On 08/30/2012 07:51 PM, Alexey Kazakov wrote:
To be honest, I ignore Jenkins emails. My experience shows me that
95%
of these problems are problems of hudson build(s)/environment/... or
problems which I'm already aware of.
So you mean that Jenkins jobs are not
useful for you? If there are
problems with builds/environment, then please open Jiras to explain what
are those problems and how to solve them. Jenkins is meant to be helpful
for you.
I don't think that it's a good idea to rely on build or QE team to
provide feedback on your component build, we can't spend all our time to
look at status of Jenkins job and warn X or Y about an issue in a build.
It should be the reverse process: component developer look at there CI
job, and when they see an error is caused by Jenkins environment, they
push it up to build through a Jira. Or if not exclusively, it should be
both process: Nick/Denis/I monitor Jenkins to see whether there are some
issues, and at the same time the component team look at their jobs to
see whether there are some issues. We'll probably reduce the feedback
loop that way.
Instead of rely on the hudson builds I regularly run tycho build
locally. It safes our time. So instead of investigating what is going
on with hudson builds every time I got Jenkis email we (CDI/Seam/JSF
team) can spend time on development running local tycho build every
other day to make sure everything is ok on our side.
I understand that Jenkins has
a long time-to-feedback currently. That's
something that is currently discussed to get way shorter feedback, so it
will be more helpful. Doing local build is a very good habit, and is
often more performant than waiting for CI feedback. But ignoring
feedback for CI is just like ignoring a colleague who says "I can't
build", there is a reason, which may be caused by anyone, and it's
necessary to determine what is the reason and how to solve it.
We mentioned these problems in JIRA and in emails but the hudson
builds is still unstable.
We are working and make it "faster" and better
isolation between
components will make it more stable. But I can understand how current
problems led you to ignore Jenkins mails, that's what we need to fix ;)
--
Mickael Istria
Eclipse developer at JBoss, by Red Hat <http://www.jboss.org/tools>
My blog <http://mickaelistria.wordpress.com> - My Tweets
<http://twitter.com/mickaelistria>
8:22 p.m.
It seems to be fine now, at least common build is not failing anymore.
Denis
On 08/31/2012 12:10 AM, Max Rydahl Andersen wrote:
have this change been done on our jenkins instance ?
/max
On 30 Aug 2012, at 23:28, Denis Golovin <dgolovin(a)exadel.com> wrote:
> That's what I found. The problem is in vncserver. Latest vnc in
> combination with gnome manager makes tests waiting forever in UI thread
> when importing projects and use JobMtils.waitForIdle(). It turned out to
> make the tests move ahead I needed to open vnc display in viewer and
> sent any event from keyboard of from mouse.
>
> the fix was replacing in ~/.vnc/xstartup
>
> gnome-session &
>
> to
>
> metacity --replace &
>
> and tests are fine again.
>
> I'm not sure but that might be the case.
>
> Denis
>
> On 08/30/2012 01:04 PM, Nick Boldt wrote:
>> Not sure... you tell me:
>>
>>
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstools-3.3_stabl...
>> (3.3.x branch)
>>
>>
https://jenkins.mw.lab.eng.bos.redhat.com/hudson/job/jbosstools-3.3_trunk...
>> (trunk)
>>
>>
>> On 08/30/2012 03:02 PM, Alexey Kazakov wrote:
>>> On 08/30/2012 11:41 AM, Nick Boldt wrote:
>>>> If you spread the 25 components out in approximate order, assuming
>>>> that no two run in parallel (which is of course not the case but makes
>>>> it easier to estimate times) then a fresh CDI build is over 9 hours
>>>> after a fresh usage, tests, or common build. Since CDI has over an
>>>> hour of tests in it, Seam is therefore close to 12 hours down the stack.
>>> An hour for CDI tests? Why so long? It takes ~ 11 minutes (about 1000
>>> JUnit tests) with Tycho on my laptop.
> _______________________________________________
> jbosstools-dev mailing list
> jbosstools-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jbosstools-dev
4255
days inactive
4256
days old
jbosstools-dev@lists.jboss.org
8 comments
5 participants
participants (5)
-
Alexey Kazakov -
Denis Golovin -
Max Rydahl Andersen -
Mickael Istria -
Nick Boldt