[hibernate-dev] ci.hibernate.org : announcing distributed cache for maven artifacts

Tue Jan 16 16:30:04 EST 2018

Yes I did it for Gradle too, sorry. The `/efs-maven-artifacts` is the
guilty mount point.

I don't know any quick solutions for the various concerns you all
raised, so I'll roll this back tonight.

It's good to know that it's not too hard to have a shared FS between
these machines; needs better planning though.

Thanks,
Sanne

On 16 January 2018 at 19:41, Steve Ebersole <steve at hibernate.org> wrote:
> Did you happen to do the same for Gradle caches?
>
> Some jobs are failing:
>
>
> * What went wrong:
> Could not resolve all dependencies for configuration
> ':buildSrc:runtimeClasspath'.
>> Timeout waiting to lock artifact cache
>> (/efs-maven-artifacts/.gradle/caches/modules-2). It is currently in use by
>> another Gradle instance.
>   Owner PID: 1423
>   Our PID: 10249
>   Owner Operation: resolve configuration ':classpath'
>   Our operation:
>   Lock file: /efs-maven-artifacts/.gradle/caches/modules-2/modules-2.lock
>
>
>
> On Mon, Jan 15, 2018 at 5:06 AM Yoann Rodiere <yoann at hibernate.org> wrote:
>>
>> > We should reconfigure those to not "install" - that's actually a bad
>> > habit, legacy from Maven 2 times - people nowadays recommend using
>> > "mvn clean verify", especially on CI environments.
>>
>> I could not agree more, that would be cleaner, but that's not possible.
>> And
>> believe me, I tried hard. Last time I checked, some of the plugins we use
>> with dynamic dependency resolution would ignore the artifacts being built,
>> and would always fetch the artifacts from the Maven repos (for SNAPSHOTs,
>> they would end up using nightlies).
>> I'm not talking about when we use standard maven markup to declare
>> dependencies, but when the plugin itself has to fetch dependencies
>> "dynamically", which happens when we setup a WildFly server with our own
>> modules in particular. See maven-dependency-plugin's "artifactItems"
>> configuration.
>>
>>
>>
>> On Mon, 15 Jan 2018 at 11:29 Sanne Grinovero <sanne at hibernate.org> wrote:
>>
>> > On 15 January 2018 at 08:42, Yoann Rodiere <yoann at hibernate.org> wrote:
>> > > Thanks Sanne !
>> > >
>> > > I have one question...
>> > >
>> > >> Please never rely on this as "storage": it's just meant as cache and
>> > >> we reserve the right to wipe it all out at any time.
>> > >
>> > > I gather you say that so that we don't try to "release" artifacts into
>> > this
>> > > cache? But temporary storage for the duration of one build will still
>> > > be
>> > > safe?
>> > >
>> > > Because our builds obviously rely on the local repository for
>> > > short-term
>> > > storage (for the duration of the build). For example the dependencies
>> > > are
>> > > only checked and downloaded if necessary at the beginning of the
>> > > build,
>> > and
>> > > then are expected to exist in the local repository until the build
>> > > stops.
>> > > Another example: our WildFly modules are first built and installed in
>> > > the
>> > > "modules" subproject, and later "fetched" from the local repository in
>> > the
>> > > "integrationtest/wildfly" subproject.
>> > >
>> > > If we were to clear the cache during a build, things would probably go
>> > > wrong. Worse, if two parallel builds were to install the same
>> > > artifacts
>> > > (e.g. hibernate-search-engine version 5.9.0-SNAPSHOT), we would run
>> > > the
>> > risk
>> > > of testing the wrong "version" of this artifact in one of the
>> > > builds...
>> >
>> > SNAPSHOT being installed are indeed a problem, e.g the PR testing jobs
>> > could conflict with the regular master jobs.
>> > We should reconfigure those to not "install" - that's actually a bad
>> > habit, legacy from Maven 2 times - people nowadays recommend using
>> > "mvn clean verify", especially on CI environments.
>> >
>> > I agree about the perils of clearing the cache during in-progress builds
>> > too.
>> >
>> > I just meant to warn that we don't have any backup plan in place, and
>> > I do plan to just wipe the whole thing occasionally:
>> >  - when we have any direct need, e.g. currupted downloads
>> >  - when it gets too large
>> >  - if it gets too expensive
>> >  - regularly, just to "practice" that everything works with an empty
>> > cache
>> >
>> > Also our "disaster recovery" plan to rebuild all infrastructure will
>> > always assume it's ok to reboot with having this file system empty.
>> >
>> > Thanks,
>> > Sanne
>> >
>> > >
>> > >
>> > > On Sun, 14 Jan 2018 at 01:18 Sanne Grinovero <sanne at hibernate.org>
>> > wrote:
>> > >>
>> > >> Hi all,
>> > >>
>> > >> while the new build machines are fast, some of you pointed out we're
>> > >> now spending a relative high amount of time downloading maven
>> > >> dependencies, this problem being compounded by the fact we "nuke"
>> > >> idle
>> > >> slaves shortly after they become idle.
>> > >>
>> > >> I just spent the day testing a distributed file system, and it's now
>> > >> running in "production".
>> > >> It's used exclusively to store the Gradle and Maven caches. This is
>> > >> stateful and independent from the lifecycle of individual slave
>> > >> nodes.
>> > >>
>> > >> Unfortunately this solution is not viable for Docker images, so while
>> > >> I experimented with the idea I backed off from moving the docker
>> > >> storage graph to a similar device. Please don't waste time trying
>> > >> that
>> > >> w/o carefully reading the Docker documentation or talking with me :)
>> > >> Also, beyond correctness of storage semantics, it's likely far less
>> > >> efficient for Docker.
>> > >>
>> > >> To learn more about our new cache:
>> > >>  -
>> > >>
>> >
>> > https://github.com/hibernate/ci.hibernate.org/commit/dc6e0a4bd09fb3ae6347081243b4fb796a219f90
>> > >>  - https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html
>> > >>
>> > >> I'd add that - because of other IO tuning in place - writes might
>> > >> appear out of order to other nodes, and conflicts are not handled.
>> > >> Shouldn't be a problem since snapshots now have timestamps, but this
>> > >> might be something to keep in mind.
>> > >>
>> > >> N.B.
>> > >> Please never rely on this as "storage": it's just meant as cache and
>> > >> we reserve the right to wipe it all out at any time.
>> > >>
>> > >> Thanks,
>> > >> Sanne
>> > >> _______________________________________________
>> > >> hibernate-dev mailing list
>> > >> hibernate-dev at lists.jboss.org
>> > >> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>> > >
>> > >
>> > >
>> > > --
>> > > Yoann Rodiere
>> > > yoann at hibernate.org / yrodiere at redhat.com
>> > > Software Engineer
>> > > Hibernate NoORM team
>> >
>>
>>
>> --
>> Yoann Rodiere
>> yoann at hibernate.org / yrodiere at redhat.com
>> Software Engineer
>> Hibernate NoORM team
>> _______________________________________________
>> hibernate-dev mailing list
>> hibernate-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hibernate-dev