Re: [hibernate-dev] ci.hibernate.org : announcing distributed cache for maven artifacts

Did you happen to do the same for Gradle caches? Some jobs are failing: * What went wrong: Could not resolve all dependencies for configuration ':buildSrc:runtimeClasspath'. > Timeout waiting to lock artifact cache > (/efs-maven-artifacts/.gradle/caches/modules-2). It is currently in use by > another Gradle instance. Owner PID: 1423 Our PID: 10249 Owner Operation: resolve configuration ':classpath' Our operation: Lock file: /efs-maven-artifacts/.gradle/caches/modules-2/modules-2.lock On Mon, Jan 15, 2018 at 5:06 AM Yoann Rodiere <yoann(a)hibernate.org&gt; wrote: > > > We should reconfigure those to not "install" - that's actually a bad > > habit, legacy from Maven 2 times - people nowadays recommend using > > "mvn clean verify", especially on CI environments. > > I could not agree more, that would be cleaner, but that's not possible. > And > believe me, I tried hard. Last time I checked, some of the plugins we use > with dynamic dependency resolution would ignore the artifacts being built, > and would always fetch the artifacts from the Maven repos (for SNAPSHOTs, > they would end up using nightlies). > I'm not talking about when we use standard maven markup to declare > dependencies, but when the plugin itself has to fetch dependencies > "dynamically", which happens when we setup a WildFly server with our own > modules in particular. See maven-dependency-plugin's "artifactItems" > configuration. > > > > On Mon, 15 Jan 2018 at 11:29 Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: > > > On 15 January 2018 at 08:42, Yoann Rodiere <yoann(a)hibernate.org&gt; wrote: > > > Thanks Sanne ! > > > > > > I have one question... > > > > > >> Please never rely on this as "storage": it's just meant as cache and > > >> we reserve the right to wipe it all out at any time. > > > > > > I gather you say that so that we don't try to "release" artifacts into > > this > > > cache? But temporary storage for the duration of one build will still > > > be > > > safe? > > > > > > Because our builds obviously rely on the local repository for > > > short-term > > > storage (for the duration of the build). For example the dependencies > > > are > > > only checked and downloaded if necessary at the beginning of the > > > build, > > and > > > then are expected to exist in the local repository until the build > > > stops. > > > Another example: our WildFly modules are first built and installed in > > > the > > > "modules" subproject, and later "fetched" from the local repository in > > the > > > "integrationtest/wildfly" subproject. > > > > > > If we were to clear the cache during a build, things would probably go > > > wrong. Worse, if two parallel builds were to install the same > > > artifacts > > > (e.g. hibernate-search-engine version 5.9.0-SNAPSHOT), we would run > > > the > > risk > > > of testing the wrong "version" of this artifact in one of the > > > builds... > > > > SNAPSHOT being installed are indeed a problem, e.g the PR testing jobs > > could conflict with the regular master jobs. > > We should reconfigure those to not "install" - that's actually a bad > > habit, legacy from Maven 2 times - people nowadays recommend using > > "mvn clean verify", especially on CI environments. > > > > I agree about the perils of clearing the cache during in-progress builds > > too. > > > > I just meant to warn that we don't have any backup plan in place, and > > I do plan to just wipe the whole thing occasionally: > > - when we have any direct need, e.g. currupted downloads > > - when it gets too large > > - if it gets too expensive > > - regularly, just to "practice" that everything works with an empty > > cache > > > > Also our "disaster recovery" plan to rebuild all infrastructure will > > always assume it's ok to reboot with having this file system empty. > > > > Thanks, > > Sanne > > > > > > > > > > > On Sun, 14 Jan 2018 at 01:18 Sanne Grinovero <sanne(a)hibernate.org&gt; > > wrote: > > >> > > >> Hi all, > > >> > > >> while the new build machines are fast, some of you pointed out we're > > >> now spending a relative high amount of time downloading maven > > >> dependencies, this problem being compounded by the fact we "nuke" > > >> idle > > >> slaves shortly after they become idle. > > >> > > >> I just spent the day testing a distributed file system, and it's now > > >> running in "production". > > >> It's used exclusively to store the Gradle and Maven caches. This is > > >> stateful and independent from the lifecycle of individual slave > > >> nodes. > > >> > > >> Unfortunately this solution is not viable for Docker images, so while > > >> I experimented with the idea I backed off from moving the docker > > >> storage graph to a similar device. Please don't waste time trying > > >> that > > >> w/o carefully reading the Docker documentation or talking with me :) > > >> Also, beyond correctness of storage semantics, it's likely far less > > >> efficient for Docker. > > >> > > >> To learn more about our new cache: > > >> - > > >> > > > > https://github.com/hibernate/ci.hibernate.org/commit/dc6e0a4bd09fb3ae6347... > > >> - https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html > > >> > > >> I'd add that - because of other IO tuning in place - writes might > > >> appear out of order to other nodes, and conflicts are not handled. > > >> Shouldn't be a problem since snapshots now have timestamps, but this > > >> might be something to keep in mind. > > >> > > >> N.B. > > >> Please never rely on this as "storage": it's just meant as cache and > > >> we reserve the right to wipe it all out at any time. > > >> > > >> Thanks, > > >> Sanne > > >> _______________________________________________ > > >> hibernate-dev mailing list > > >> hibernate-dev(a)lists.jboss.org > > >> https://lists.jboss.org/mailman/listinfo/hibernate-dev > > > > > > > > > > > > -- > > > Yoann Rodiere > > > yoann(a)hibernate.org / yrodiere(a)redhat.com > > > Software Engineer > > > Hibernate NoORM team > > > > > -- > Yoann Rodiere > yoann(a)hibernate.org / yrodiere(a)redhat.com > Software Engineer > Hibernate NoORM team > _______________________________________________ > hibernate-dev mailing list > hibernate-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev