[hibernate-dev] [OGM] CI jobs hang (Gunnar Morling)

Wed Apr 2 11:51:54 EDT 2014

Hi Gunnar,

here a small preview of what I am working with at the moment.

Everything is still in beta so technically everything can change in future.

I am working mainly on 2 fronts:

1) allow in-container compilation.
This should help to isolate concurrent build from the network point of
view(no ports clashing) and using iptables from the network altogether(no
risks of spontaneous clusters formation or any related problem). On top of
that switching from one version to another of jdk should be easier since
the complexity of configuring them is hidden inside the containers and no
longer handled an the main host level.

2) Using Docker containers to provide dedicate instances of  external
services
With Docker I can start an instance of Mongo or Couch, that lives in it's
own ip address and that is tied to a specific build job and automatically
disposed at the end of the job. Again, the complexity of the configuration
lives inside the containers, so once that is done the solution is easily
reusable by every devloper, without asking him to manually download and
configure the application.

At jenkins level at the moment I am running invocations similar to this:

IP_DOCKER_HOST=$(ip -f inet -o addr show docker0 | cut -d ' '  -f 7 | cut
-d '/' -f 1) \
PRJ_FOLDER_HOST="$WORKSPACE" \
PRJ_FOLDER_CONTAINER="/data/maven_build_folder" \
MVN_M2_HOST="/home/jenkins/.m2" \
MVN_M2_CONTAINER="/root/.m2" ; \
docker run -v "$PRJ_FOLDER_HOST":"$PRJ_FOLDER_CONTAINER" \
     -v "$MVN_M2_HOST":"$MVN_M2_CONTAINER" \
     -w "$PRJ_FOLDER_CONTAINER" \
      --privileged openjdk7_maven:iptables \
     clean verify  -s $MVN_M2_CONTAINER/settings-search-release.xml
-DskipTests=false -Dcheckstyle.skip=true -DskipDocs=true -DuseDocker=true
-DuseExternalMongoDb -Ddocker.daemon.ip=$IP_DOCKER_HOST

That is a little scary but most ov it is just configuration to share
folders from the host to the container.

See also a couple of screenshots that show how I can have 5 concurrent
running job, building OGM, each one starting its own private disposable
instance of Couch and Mongo.

http://imgur.com/kWZVpLh&gRpts1U#0

http://imgur.com/kWZVpLh&gRpts1U#1

I am right now working on Gradle based build to see if they can also fit
well within this approach.

cheers

paolo

On Tue, Apr 1, 2014 at 2:11 PM, Gunnar Morling <gunnar at hibernate.org> wrote:

> Hi Paolo,
>
> Thanks for your reply. The thing is that these jobs used to work until
> recently, so I'm wondering which change could be causing the problem. Maybe
> a recent update to Jenkins or one of the plug-ins?
>
> Unfortunately we don't have the config audit plug-in anymore which
> provides a history at least of changes to job configs. @Davide, I think I
> had installed it once but I guess it got lost when rebuilding the instance
> via Puppet? Any chance you could add this plug-in permanently?
>
> @Paolo: Would be very interested to learn more about the Docker stuff
> you're working on once you got something :)
>
> --Gunnar
>
>
>
>
>
> 2014-04-01 15:02 GMT+02:00 Paolo Antinori <paolo.antinori at gmail.com>:
>
>> Hi Gunnar
>>
>> I am working on related activities on a replica of ci.hibernate.orgwhere I
>> am playing with Docker and trying to see how well it could fit the the
>> project build needs.
>>
>> When running OGM build job on this replica server, I have noticed the same
>> problem.
>>
>> I think that the problem is related to Axis plugin, like you were pointing
>> out.
>>
>> I have also noticed this:
>>
>> http://ci.hibernate.org/job/hibernate-ogm-master/106/consoleText
>>
>> That is the link for the plain text version of the logs.
>>
>> Not sure if the message you find there:
>>
>> No such file:
>> /var/lib/jenkins/jobs/hibernate-ogm-master/builds/2014-04-01_07-01-01/log
>>
>> Is the blocking problem or just a side-effect of something else.
>>
>> I hope to be able to show soon a Docker based approach to possibly bypass
>> totally this kind of problems.
>>
>> paolo
>>
>>
>> On Tue, Apr 1, 2014 at 12:46 PM, <hibernate-dev-request at lists.jboss.org
>> >wrote:
>>
>> > Send hibernate-dev mailing list submissions to
>> >         hibernate-dev at lists.jboss.org
>> >
>> > To subscribe or unsubscribe via the World Wide Web, visit
>> >         https://lists.jboss.org/mailman/listinfo/hibernate-dev
>> > or, via email, send a message with subject or body 'help' to
>> >         hibernate-dev-request at lists.jboss.org
>> >
>> > You can reach the person managing the list at
>> >         hibernate-dev-owner at lists.jboss.org
>> >
>> > When replying, please edit your Subject line so it is more specific
>> > than "Re: Contents of hibernate-dev digest..."
>> >
>> > Today's Topics:
>> >
>> >    1. [OGM] CI jobs hang (Gunnar Morling)
>> >    2. Re: Session and carrying 3rd party state (Gunnar Morling)
>> >    3. Re: Session and carrying 3rd party state (Gunnar Morling)
>> >
>> >
>> > ---------- Forwarded message ----------
>> > From: Gunnar Morling <gunnar at hibernate.org>
>> > To: "hibernate-dev at lists.jboss.org" <hibernate-dev at lists.jboss.org>
>> > Cc:
>> > Date: Tue, 1 Apr 2014 09:20:28 +0200
>> > Subject: [hibernate-dev] [OGM] CI jobs hang
>> > Hi,
>> >
>> > For some reason, the CI jobs for OGM can't be executed (all jobs,
>> master,
>> > java8, PR seem affected) . They hang forever in an initial state, i.e.
>> > there is no console output and also the sub-jobs are not dispatched
>> (these
>> > are matrix projects). Worse, they can't even be canceled (trying to do
>> so
>> > has no effect), only a server restart helps.
>> >
>> > Have there been any recent changes to the job (or server) config which
>> may
>> > cause this? I found
>> https://issues.jenkins-ci.org/browse/JENKINS-9688which
>> > indicates, that the jobs may be waiting eternally for a resource which
>> they
>> > can't acquire (I don't see any resources requested in the job config,
>> but I
>> > vaguely remember something around JGroups channels).
>> >
>> > Does anyone know what's going on here?
>> >
>> > --Gunnar
>> >
>> >
>> >
>> _______________________________________________
>> hibernate-dev mailing list
>> hibernate-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>
>
>