[hibernate-dev] ci.hibernate.org : announcing distributed cache for maven artifacts

[Sanne Grinovero]

Hi all,

while the new build machines are fast, some of you pointed out we're
now spending a relative high amount of time downloading maven
dependencies, this problem being compounded by the fact we "nuke" idle
slaves shortly after they become idle.

I just spent the day testing a distributed file system, and it's now
running in "production".
It's used exclusively to store the Gradle and Maven caches. This is
stateful and independent from the lifecycle of individual slave nodes.

Unfortunately this solution is not viable for Docker images, so while
I experimented with the idea I backed off from moving the docker
storage graph to a similar device. Please don't waste time trying that
w/o carefully reading the Docker documentation or talking with me :)
Also, beyond correctness of storage semantics, it's likely far less
efficient for Docker.

To learn more about our new cache:
 - https://github.com/hibernate/ci.hibernate.org/commit/dc6e0a4bd09fb3ae6347081243b4fb796a219f90
 - https://docs.aws.amazon.com/efs/latest/ug/how-it-works.html

I'd add that - because of other IO tuning in place - writes might
appear out of order to other nodes, and conflicts are not handled.
Shouldn't be a problem since snapshots now have timestamps, but this
might be something to keep in mind.

N.B.
Please never rely on this as "storage": it's just meant as cache and
we reserve the right to wipe it all out at any time.

Thanks,
Sanne