Re: [hibernate-dev] ci.hibernate.org : announcing distributed cache for maven artifacts

Thanks Sanne ! I have one question... I gather you say that so that we don't try to "release" artifacts into this cache? But temporary storage for the duration of one build will still be safe? Because our builds obviously rely on the local repository for short-term storage (for the duration of the build). For example the dependencies are only checked and downloaded if necessary at the beginning of the build, and then are expected to exist in the local repository until the build stops. Another example: our WildFly modules are first built and installed in the "modules" subproject, and later "fetched" from the local repository in the "integrationtest/wildfly" subproject. If we were to clear the cache during a build, things would probably go wrong. Worse, if two parallel builds were to install the same artifacts (e.g. hibernate-search-engine version 5.9.0-SNAPSHOT), we would run the risk of testing the wrong "version" of this artifact in one of the builds... On Sun, 14 Jan 2018 at 01:18 Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: > Hi all, > > while the new build machines are fast, some of you pointed out we're > now spending a relative high amount of time downloading maven > dependencies, this problem being compounded by the fact we "nuke" idle > slaves shortly after they become idle. > > I just spent the day testing a distributed file system, and it's now > running in "production". > It's used exclusively to store the Gradle and Maven caches. This is > stateful and independent from the lifecycle of individual slave nodes. > > Unfortunately this solution is not viable for Docker images, so while > I experimented with the idea I backed off from moving the docker > storage graph to a similar device. Please don't waste time trying that > w/o carefully reading the Docker documentation or talking with me :) > Also, beyond correctness of storage semantics, it's likely far less > efficient for Docker. > > To learn more about our new cache: > - > https://github.com/hibernate/ci.hibernate.org/commit/dc6e0a4bd09fb3ae6347... > - https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html > > I'd add that - because of other IO tuning in place - writes might > appear out of order to other nodes, and conflicts are not handled. > Shouldn't be a problem since snapshots now have timestamps, but this > might be something to keep in mind. > > N.B. > > Thanks, > Sanne > _______________________________________________ > hibernate-dev mailing list > hibernate-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev > -- Yoann Rodiere yoann(a)hibernate.org / yrodiere(a)redhat.com Software Engineer Hibernate NoORM team