Awesome Sanne! Great work. Anything you need us to do to our jobs? On Thu, Jan 4, 2018, 5:20 PM Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: > Hi all, > > we're having shiny new boxes running CI: more secure, way faster and > less "out of disk space" prolems I hope. > > # Slaves > > Slaves have been rebuilt from scratch: > - from Fedora 25 to Fedora 27 > - NVMe disks for all storage, including databases, JDKs, dependency > stores, indexes and journals > - Now using C5 instances to benefit from Amazon's new "Nitro" engines [1] > - hardware offloading of network operations by enabling ENA [2] > - NVMe drives also using provisioned IO > > This took a bit of unexpected low level work as .. Fedora images don't > support ENA yet so I had to create a custom Fedora re-distribution AMI > first, it wasn't possible to simply compile the kernel modules for the > standard Fedora images. These features are expected to come in future > Fedora Cloud images but I didn't want to wait so made our own :) [3] > > # Cloud scaling > > Idle slaves will self-terminate after some timeout (currently 30m). > When there are many jobs queueing up, more slaves (up to 5) will > automatically start. > > If you're the first to trigger a build you'll have to be patient, as > it's possible after some quiet time (after the night?) all slaves are > gone; the system will boot up new ones automatically ASAP but this > initial boot takes some extra couple of minutes. > > # Master node > > Well, security patching mostly, but also finally figured out how to > workaround the bugs which were preventing us to upgrade Jenkins. > > So now Jenkins is upgraded to latest, including *all plugins*. It > seems to work but let's keep an eye on it, those plugins are not all > maintained at the quality one would expect. > > In particular attempting to change EC2 configuration properties will > now trigger a super annoying NPE [4]; either don't make further > changes or resort to XML editing of the configuration. > > # Next > > I'm not entirely done; eventually I'd like to convert our master node > to ENA/C5/NVMe as well - especially to be able to move all master and > slaves into the same physical cluster - but I'll stop now and get back > to Java so you all get a chance to identify problems caused by the new > slaves before I cause more trouble.. > > Thanks, > Sanne > > 1 - > https://www.theregister.co.uk/2017/11/29/aws_reveals_nitro_ architecture_bare_metal_ec2_guard_duty_security_tool/ > 2 - > https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ enhanced-networking-ena.html > 3 - https://pagure.io/atomic-wg/issue/271 > 4 - https://issues.jenkins-ci.org/browse/JENKINS-46856 > _______________________________________________ > hibernate-dev mailing list > hibernate-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev > _______________________________________________ hibernate-dev mailing list hibernate-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/hibernate-dev

Great, thanks for all the work! Now that we have on-demand slave spawning, maybe we could get rid of our "hack" consisting in assigning 5 slots to each slave and a weight of 3 to each job? I would expect the website and release jobs to rarely wait in the queue, and if they do we can always set up a specific "priority queue" for those jobs, with a dedicated slave pool. Just asking for this because last time I checked, it was not possible to assign weight to jobs defined as Jenkins pipelines. So these jobs ended up with a weight of 1, and we ended up running multiple instances of those on the same slave... which is obviously not good. I can do the boring job editing work on each and every job, I'm just asking if it is seems ok to you... ?

Yoann Rodière Hibernate NoORM Team yoann(a)hibernate.org On 5 January 2018 at 00:52, Steve Ebersole <steve(a)hibernate.org&gt; wrote: > > Awesome Sanne! Great work. > > Anything you need us to do to our jobs? > > On Thu, Jan 4, 2018, 5:20 PM Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: > > > Hi all, > > > > we're having shiny new boxes running CI: more secure, way faster and > > less "out of disk space" prolems I hope. > > > > # Slaves > > > > Slaves have been rebuilt from scratch: > > - from Fedora 25 to Fedora 27 > > - NVMe disks for all storage, including databases, JDKs, dependency > > stores, indexes and journals > > - Now using C5 instances to benefit from Amazon's new "Nitro" engines > > [1] > > - hardware offloading of network operations by enabling ENA [2] > > - NVMe drives also using provisioned IO > > > > This took a bit of unexpected low level work as .. Fedora images don't > > support ENA yet so I had to create a custom Fedora re-distribution AMI > > first, it wasn't possible to simply compile the kernel modules for the > > standard Fedora images. These features are expected to come in future > > Fedora Cloud images but I didn't want to wait so made our own :) [3] > > > > # Cloud scaling > > > > Idle slaves will self-terminate after some timeout (currently 30m). > > When there are many jobs queueing up, more slaves (up to 5) will > > automatically start. > > > > If you're the first to trigger a build you'll have to be patient, as > > it's possible after some quiet time (after the night?) all slaves are > > gone; the system will boot up new ones automatically ASAP but this > > initial boot takes some extra couple of minutes. > > > > # Master node > > > > Well, security patching mostly, but also finally figured out how to > > workaround the bugs which were preventing us to upgrade Jenkins. > > > > So now Jenkins is upgraded to latest, including *all plugins*. It > > seems to work but let's keep an eye on it, those plugins are not all > > maintained at the quality one would expect. > > > > In particular attempting to change EC2 configuration properties will > > now trigger a super annoying NPE [4]; either don't make further > > changes or resort to XML editing of the configuration. > > > > # Next > > > > I'm not entirely done; eventually I'd like to convert our master node > > to ENA/C5/NVMe as well - especially to be able to move all master and > > slaves into the same physical cluster - but I'll stop now and get back > > to Java so you all get a chance to identify problems caused by the new > > slaves before I cause more trouble.. > > > > Thanks, > > Sanne > > > > 1 - > > > > https://www.theregister.co.uk/2017/11/29/aws_reveals_nitro_architecture_b... > > 2 - > > > > https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-e... > > 3 - https://pagure.io/atomic-wg/issue/271 > > 4 - https://issues.jenkins-ci.org/browse/JENKINS-46856 > > _______________________________________________ > > hibernate-dev mailing list > > hibernate-dev(a)lists.jboss.org > > https://lists.jboss.org/mailman/listinfo/hibernate-dev > > > _______________________________________________ > hibernate-dev mailing list > hibernate-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/hibernate-dev

I went to manually kick off the main ORM job, but saw that you already had - however it had failed with GC/memory problems[1]. I kicked off a new run... [1] http://ci.hibernate.org/job/hibernate-orm-master-h2-main/951/console On Fri, Jan 5, 2018 at 4:59 AM Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: > On 4 January 2018 at 23:52, Steve Ebersole <steve(a)hibernate.org&gt; wrote: > > Awesome Sanne! Great work. > > > > Anything you need us to do to our jobs? > > No changes *should* be needed. It would help me if you could all > manually trigger the jobs you consider important and highlight > suspucious problems so that we get awareness of regressions in short > time. > > I've already triggered some ~20 jobs last night; saw no problems so > far but haven't tested any release yet, nor website related tasks. > > Thanks, > Sanne >

On 5 January 2018 at 12:28, Steve Ebersole <steve(a)hibernate.org&gt; wrote: > FWIW... I do not know the rules about how these slaves spin up, but in the > 10+ minutes since I kicked off that job it is still waiting in queue. When there are no slaves it might take some extra minutes; on top of that I was manually killing some leftover machines from yesterday's night experiments, so maybe I bothered it in some way. Let's keep an eye on it, if it happens regularly we'll see what can be done. I'll likely want to keep a slave "always on".. > And there is actually a job (Debezium Deploy Snapshots) in front of it that has > been waiting over 3.5 hours That was my fault, thanks for spotting it! (the job was misconfigured, fixed now). > On Fri, Jan 5, 2018 at 6:20 AM Steve Ebersole <steve(a)hibernate.org&gt; wrote: >> >> I went to manually kick off the main ORM job, but saw that you already had >> - however it had failed with GC/memory problems[1]. I kicked off a new >> run... These boxes have 4 core and 8GB RAM heach. We can probably use larger heaps: I've reconfigured the gradle and Maven environment options to allow 4GB of heap, and kicked a new ORM build.

I have no idea what GitBlamer is. Never heard of it

On Fri, Jan 5, 2018, 7:38 AM Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: > > On 5 January 2018 at 13:12, Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: > > On 5 January 2018 at 12:28, Steve Ebersole <steve(a)hibernate.org&gt; wrote: > >> FWIW... I do not know the rules about how these slaves spin up, but in > >> the > >> 10+ minutes since I kicked off that job it is still waiting in queue. > > > > When there are no slaves it might take some extra minutes; on top of > > that I was manually killing some leftover machines from yesterday's > > night experiments, so maybe I bothered it in some way. > > > > Let's keep an eye on it, if it happens regularly we'll see what can be > > done. I'll likely want to keep a slave "always on".. > > > >> And there is actually a job (Debezium Deploy Snapshots) in front of it > >> that has > >> been waiting over 3.5 hours > > > > That was my fault, thanks for spotting it! (the job was misconfigured, > > fixed now). > > > >> On Fri, Jan 5, 2018 at 6:20 AM Steve Ebersole <steve(a)hibernate.org&gt; > >> wrote: > >>> > >>> I went to manually kick off the main ORM job, but saw that you already > >>> had > >>> - however it had failed with GC/memory problems[1]. I kicked off a > >>> new > >>> run... > > > > These boxes have 4 core and 8GB RAM heach. We can probably use larger > > heaps: I've reconfigured the gradle and Maven environment options to > > allow 4GB of heap, and kicked a new ORM build. > > The Gradle build completed fine now, in just 12 minutes. > However then the job gets stuck on: > > <Git Blamer> Using GitBlamer to create author and commit > information for all warnings. > > It's been busy 15 minutes already at this point - so taking longer > than the build and tests - and still going.. > What is it? is it worth it? something wrong with it? > > - http://ci.hibernate.org/job/hibernate-orm-master-h2-main/953/console > > Thanks, > Sanne

TBH, I'm ok with just dropping the TODO collection as a part of the Jenkins jobs.

On Fri, Jan 5, 2018 at 8:56 AM Sanne Grinovero <sanne(a)hibernate.org&gt; wrote: > > On 5 January 2018 at 14:07, Steve Ebersole <steve(a)hibernate.org&gt; wrote: > > I have no idea what GitBlamer is. Never heard of it > > I figured it out; it's implicitly (by default) invoked by the job > tasks of finding "TODO"'s and similar markers in the project, > to add "blame" information to the final report. > > So for each and every warning the report would produce, it will dig > into the git history of the project to figure out who introduced the > marker. I disabled this "blame" process, I hope that's allright - it > worked fine now, producing a full build in 15 minutes: > > - http://ci.hibernate.org/job/hibernate-orm-master-h2-main/955/console > > Since we didn't see this problem before I guess it's a new "feature" > caused by me by upgrading the plugins. > I disabled that "feature" globally as I don't think it's reasonable > for any of our projects.. > > Thanks, > Sanne > > > > > > > On Fri, Jan 5, 2018, 7:38 AM Sanne Grinovero <sanne(a)hibernate.org&gt; > > wrote: > >> > >> On 5 January 2018 at 13:12, Sanne Grinovero <sanne(a)hibernate.org&gt; > >> wrote: > >> > On 5 January 2018 at 12:28, Steve Ebersole <steve(a)hibernate.org&gt; > >> > wrote: > >> >> FWIW... I do not know the rules about how these slaves spin up, but > >> >> in > >> >> the > >> >> 10+ minutes since I kicked off that job it is still waiting in > >> >> queue. > >> > > >> > When there are no slaves it might take some extra minutes; on top of > >> > that I was manually killing some leftover machines from yesterday's > >> > night experiments, so maybe I bothered it in some way. > >> > > >> > Let's keep an eye on it, if it happens regularly we'll see what can > >> > be > >> > done. I'll likely want to keep a slave "always on".. > >> > > >> >> And there is actually a job (Debezium Deploy Snapshots) in front of > >> >> it > >> >> that has > >> >> been waiting over 3.5 hours > >> > > >> > That was my fault, thanks for spotting it! (the job was > >> > misconfigured, > >> > fixed now). > >> > > >> >> On Fri, Jan 5, 2018 at 6:20 AM Steve Ebersole <steve(a)hibernate.org&gt; > >> >> wrote: > >> >>> > >> >>> I went to manually kick off the main ORM job, but saw that you > >> >>> already > >> >>> had > >> >>> - however it had failed with GC/memory problems[1]. I kicked off a > >> >>> new > >> >>> run... > >> > > >> > These boxes have 4 core and 8GB RAM heach. We can probably use larger > >> > heaps: I've reconfigured the gradle and Maven environment options to > >> > allow 4GB of heap, and kicked a new ORM build. > >> > >> The Gradle build completed fine now, in just 12 minutes. > >> However then the job gets stuck on: > >> > >> <Git Blamer> Using GitBlamer to create author and commit > >> information for all warnings. > >> > >> It's been busy 15 minutes already at this point - so taking longer > >> than the build and tests - and still going.. > >> What is it? is it worth it? something wrong with it? > >> > >> - http://ci.hibernate.org/job/hibernate-orm-master-h2-main/953/console > >> > >> Thanks, > >> Sanne