[hibernate-dev] New CI slaves now available!

[Steve Ebersole]

Awesome Sanne!  Great work.

Anything you need us to do to our jobs?

On Thu, Jan 4, 2018, 5:20 PM Sanne Grinovero <sanne at hibernate.org> wrote:

> Hi all,
>
> we're having shiny new boxes running CI: more secure, way faster and
> less "out of disk space" prolems I hope.
>
> # Slaves
>
> Slaves have been rebuilt from scratch:
>  - from Fedora 25 to Fedora 27
>  - NVMe disks for all storage, including databases, JDKs, dependency
> stores, indexes and journals
>  - Now using C5 instances to benefit from Amazon's new "Nitro" engines [1]
>  - hardware offloading of network operations by enabling ENA [2]
>  - NVMe drives also using provisioned IO
>
> This took a bit of unexpected low level work as .. Fedora images don't
> support ENA yet so I had to create a custom Fedora re-distribution AMI
> first, it wasn't possible to simply compile the kernel modules for the
> standard Fedora images. These features are expected to come in future
> Fedora Cloud images but I didn't want to wait so made our own :) [3]
>
> # Cloud scaling
>
> Idle slaves will self-terminate after some timeout (currently 30m).
> When there are many jobs queueing up, more slaves (up to 5) will
> automatically start.
>
> If you're the first to trigger a build you'll have to be patient, as
> it's possible after some quiet time (after the night?) all slaves are
> gone; the system will boot up new ones automatically ASAP but this
> initial boot takes some extra couple of minutes.
>
> # Master node
>
> Well, security patching mostly, but also finally figured out how to
> workaround the bugs which were preventing us to upgrade Jenkins.
>
> So now Jenkins is upgraded to latest, including *all plugins*. It
> seems to work but let's keep an eye on it, those plugins are not all
> maintained at the quality one would expect.
>
> In particular attempting to change EC2 configuration properties will
> now trigger a super annoying NPE [4]; either don't make further
> changes or resort to XML editing of the configuration.
>
> # Next
>
> I'm not entirely done; eventually I'd like to convert our master node
> to ENA/C5/NVMe as well - especially to be able to move all master and
> slaves into the same physical cluster - but I'll stop now and get back
> to Java so you all get a chance to identify problems caused by the new
> slaves before I cause more trouble..
>
> Thanks,
> Sanne
>
> 1 -
> https://www.theregister.co.uk/2017/11/29/aws_reveals_nitro_architecture_bare_metal_ec2_guard_duty_security_tool/
> 2 -
> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html
> 3 - https://pagure.io/atomic-wg/issue/271
> 4 - https://issues.jenkins-ci.org/browse/JENKINS-46856
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>