[hibernate-dev] New CI slaves now available!

[Sanne Grinovero]

Hi all,

we're having shiny new boxes running CI: more secure, way faster and
less "out of disk space" prolems I hope.

# Slaves

Slaves have been rebuilt from scratch:
 - from Fedora 25 to Fedora 27
 - NVMe disks for all storage, including databases, JDKs, dependency
stores, indexes and journals
 - Now using C5 instances to benefit from Amazon's new "Nitro" engines [1]
 - hardware offloading of network operations by enabling ENA [2]
 - NVMe drives also using provisioned IO

This took a bit of unexpected low level work as .. Fedora images don't
support ENA yet so I had to create a custom Fedora re-distribution AMI
first, it wasn't possible to simply compile the kernel modules for the
standard Fedora images. These features are expected to come in future
Fedora Cloud images but I didn't want to wait so made our own :) [3]

# Cloud scaling

Idle slaves will self-terminate after some timeout (currently 30m).
When there are many jobs queueing up, more slaves (up to 5) will
automatically start.

If you're the first to trigger a build you'll have to be patient, as
it's possible after some quiet time (after the night?) all slaves are
gone; the system will boot up new ones automatically ASAP but this
initial boot takes some extra couple of minutes.

# Master node

Well, security patching mostly, but also finally figured out how to
workaround the bugs which were preventing us to upgrade Jenkins.

So now Jenkins is upgraded to latest, including *all plugins*. It
seems to work but let's keep an eye on it, those plugins are not all
maintained at the quality one would expect.

In particular attempting to change EC2 configuration properties will
now trigger a super annoying NPE [4]; either don't make further
changes or resort to XML editing of the configuration.

# Next

I'm not entirely done; eventually I'd like to convert our master node
to ENA/C5/NVMe as well - especially to be able to move all master and
slaves into the same physical cluster - but I'll stop now and get back
to Java so you all get a chance to identify problems caused by the new
slaves before I cause more trouble..

Thanks,
Sanne

1 - https://www.theregister.co.uk/2017/11/29/aws_reveals_nitro_architecture_bare_metal_ec2_guard_duty_security_tool/
2 - https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html
3 - https://pagure.io/atomic-wg/issue/271
4 - https://issues.jenkins-ci.org/browse/JENKINS-46856