Re: [hibernate-dev] ci.hibernate.org : announcing distributed cache for maven artifacts

Version F27v17 of the slaves is running now, with NFS drive removed. sorry for the experiment :) Thanks Sanne On 16 January 2018 at 21:51, Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: > On 16 January 2018 at 21:33, Steve Ebersole <steve(a)hibernate.org&gt; wrote: >> well Gradle is used in CI environments all over the place, so it must work. >> But I think we need some different configurations in the Gradle command >> used. For example, it is highly suggested that the Gradle daemon be >> disabled in CI but I'm not sure all of our jobs actually do that. I'll look >> into that... > > I wouldn't mind having the Gradle deamon always on, if it helps we > could even pre-load it with some tuned configuration. > The only drawback I see is to make it easy to upgrade Gradle version, > in case one needs, without having to go through server configuration > scripts. > > We need strict isolation about writes in the cache though; for now > I'll disable it, not least for the concerns that Yoann and Gunnar > pointed out, then we can experiment with cool ideas more carefully. > > Funny, one would expect to know by know about the perils of a > distributed cache :) > > >> >> On Tue, Jan 16, 2018 at 3:30 PM Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: >>> >>> Yes I did it for Gradle too, sorry. The `/efs-maven-artifacts` is the >>> guilty mount point. >>> >>> I don't know any quick solutions for the various concerns you all >>> raised, so I'll roll this back tonight. >>> >>> It's good to know that it's not too hard to have a shared FS between >>> these machines; needs better planning though. >>> >>> Thanks, >>> Sanne >>> >>> On 16 January 2018 at 19:41, Steve Ebersole <steve(a)hibernate.org&gt; wrote: >>> > Did you happen to do the same for Gradle caches? >>> > >>> > Some jobs are failing: >>> > >>> > >>> > * What went wrong: >>> > Could not resolve all dependencies for configuration >>> > ':buildSrc:runtimeClasspath'. >>> >> Timeout waiting to lock artifact cache >>> >> (/efs-maven-artifacts/.gradle/caches/modules-2). It is currently in use >>> >> by >>> >> another Gradle instance. >>> > Owner PID: 1423 >>> > Our PID: 10249 >>> > Owner Operation: resolve configuration ':classpath' >>> > Our operation: >>> > Lock file: >>> > /efs-maven-artifacts/.gradle/caches/modules-2/modules-2.lock >>> > >>> > >>> > >>> > On Mon, Jan 15, 2018 at 5:06 AM Yoann Rodiere <yoann(a)hibernate.org&gt; >>> > wrote: >>> >> >>> >> > We should reconfigure those to not "install" - that's actually a bad >>> >> > habit, legacy from Maven 2 times - people nowadays recommend using >>> >> > "mvn clean verify", especially on CI environments. >>> >> >>> >> I could not agree more, that would be cleaner, but that's not possible. >>> >> And >>> >> believe me, I tried hard. Last time I checked, some of the plugins we >>> >> use >>> >> with dynamic dependency resolution would ignore the artifacts being >>> >> built, >>> >> and would always fetch the artifacts from the Maven repos (for >>> >> SNAPSHOTs, >>> >> they would end up using nightlies). >>> >> I'm not talking about when we use standard maven markup to declare >>> >> dependencies, but when the plugin itself has to fetch dependencies >>> >> "dynamically", which happens when we setup a WildFly server with our >>> >> own >>> >> modules in particular. See maven-dependency-plugin's "artifactItems" >>> >> configuration. >>> >> >>> >> >>> >> >>> >> On Mon, 15 Jan 2018 at 11:29 Sanne Grinovero <sanne(a)hibernate.org&gt; >>> >> wrote: >>> >> >>> >> > On 15 January 2018 at 08:42, Yoann Rodiere <yoann(a)hibernate.org&gt; >>> >> > wrote: >>> >> > > Thanks Sanne ! >>> >> > > >>> >> > > I have one question... >>> >> > > >>> >> > >> Please never rely on this as "storage": it's just meant as cache >>> >> > >> and >>> >> > >> we reserve the right to wipe it all out at any time. >>> >> > > >>> >> > > I gather you say that so that we don't try to "release" artifacts >>> >> > > into >>> >> > this >>> >> > > cache? But temporary storage for the duration of one build will >>> >> > > still >>> >> > > be >>> >> > > safe? >>> >> > > >>> >> > > Because our builds obviously rely on the local repository for >>> >> > > short-term >>> >> > > storage (for the duration of the build). For example the >>> >> > > dependencies >>> >> > > are >>> >> > > only checked and downloaded if necessary at the beginning of the >>> >> > > build, >>> >> > and >>> >> > > then are expected to exist in the local repository until the build >>> >> > > stops. >>> >> > > Another example: our WildFly modules are first built and installed >>> >> > > in >>> >> > > the >>> >> > > "modules" subproject, and later "fetched" from the local repository >>> >> > > in >>> >> > the >>> >> > > "integrationtest/wildfly" subproject. >>> >> > > >>> >> > > If we were to clear the cache during a build, things would probably >>> >> > > go >>> >> > > wrong. Worse, if two parallel builds were to install the same >>> >> > > artifacts >>> >> > > (e.g. hibernate-search-engine version 5.9.0-SNAPSHOT), we would run >>> >> > > the >>> >> > risk >>> >> > > of testing the wrong "version" of this artifact in one of the >>> >> > > builds... >>> >> > >>> >> > SNAPSHOT being installed are indeed a problem, e.g the PR testing >>> >> > jobs >>> >> > could conflict with the regular master jobs. >>> >> > We should reconfigure those to not "install" - that's actually a bad >>> >> > habit, legacy from Maven 2 times - people nowadays recommend using >>> >> > "mvn clean verify", especially on CI environments. >>> >> > >>> >> > I agree about the perils of clearing the cache during in-progress >>> >> > builds >>> >> > too. >>> >> > >>> >> > I just meant to warn that we don't have any backup plan in place, and >>> >> > I do plan to just wipe the whole thing occasionally: >>> >> > - when we have any direct need, e.g. currupted downloads >>> >> > - when it gets too large >>> >> > - if it gets too expensive >>> >> > - regularly, just to "practice" that everything works with an empty >>> >> > cache >>> >> > >>> >> > Also our "disaster recovery" plan to rebuild all infrastructure will >>> >> > always assume it's ok to reboot with having this file system empty. >>> >> > >>> >> > Thanks, >>> >> > Sanne >>> >> > >>> >> > > >>> >> > > >>> >> > > On Sun, 14 Jan 2018 at 01:18 Sanne Grinovero < sanne(a)hibernate.org&gt; >>> >> > wrote: >>> >> > >> >>> >> > >> Hi all, >>> >> > >> >>> >> > >> while the new build machines are fast, some of you pointed out >>> >> > >> we're >>> >> > >> now spending a relative high amount of time downloading maven >>> >> > >> dependencies, this problem being compounded by the fact we "nuke" >>> >> > >> idle >>> >> > >> slaves shortly after they become idle. >>> >> > >> >>> >> > >> I just spent the day testing a distributed file system, and it's >>> >> > >> now >>> >> > >> running in "production". >>> >> > >> It's used exclusively to store the Gradle and Maven caches. This >>> >> > >> is >>> >> > >> stateful and independent from the lifecycle of individual slave >>> >> > >> nodes. >>> >> > >> >>> >> > >> Unfortunately this solution is not viable for Docker images, so >>> >> > >> while >>> >> > >> I experimented with the idea I backed off from moving the docker >>> >> > >> storage graph to a similar device. Please don't waste time trying >>> >> > >> that >>> >> > >> w/o carefully reading the Docker documentation or talking with me >>> >> > >> :) >>> >> > >> Also, beyond correctness of storage semantics, it's likely far >>> >> > >> less >>> >> > >> efficient for Docker. >>> >> > >> >>> >> > >> To learn more about our new cache: >>> >> > >> - >>> >> > >> >>> >> > >>> >> > >>> >> > https://github.com/hibernate/ci.hibernate.org/commit/dc6e0a4bd09fb3ae6347... >>> >> > >> - https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html >>> >> > >> >>> >> > >> I'd add that - because of other IO tuning in place - writes might >>> >> > >> appear out of order to other nodes, and conflicts are not handled. >>> >> > >> Shouldn't be a problem since snapshots now have timestamps, but >>> >> > >> this >>> >> > >> might be something to keep in mind. >>> >> > >> >>> >> > >> N.B. >>> >> > >> Please never rely on this as "storage": it's just meant as cache >>> >> > >> and >>> >> > >> we reserve the right to wipe it all out at any time. >>> >> > >> >>> >> > >> Thanks, >>> >> > >> Sanne >>> >> > >> _______________________________________________ >>> >> > >> hibernate-dev mailing list >>> >> > >> hibernate-dev(a)lists.jboss.org >>> >> > >> https://lists.jboss.org/mailman/listinfo/hibernate-dev >>> >> > > >>> >> > > >>> >> > > >>> >> > > -- >>> >> > > Yoann Rodiere >>> >> > > yoann(a)hibernate.org / yrodiere(a)redhat.com >>> >> > > Software Engineer >>> >> > > Hibernate NoORM team >>> >> > >>> >> >>> >> >>> >> -- >>> >> Yoann Rodiere >>> >> yoann(a)hibernate.org / yrodiere(a)redhat.com >>> >> Software Engineer >>> >> Hibernate NoORM team >>> >> _______________________________________________ >>> >> hibernate-dev mailing list >>> >> hibernate-dev(a)lists.jboss.org >>> >> https://lists.jboss.org/mailman/listinfo/hibernate-dev