[jboss-jira] [JBoss JIRA] (SWSQE-374) When starting 40> more jenkins slaves some of them fail to start

When starting 40> more jenkins slaves some of them fail to start ----------------------------------------------------------------- Key: SWSQE-374 URL: https://issues.jboss.org/browse/SWSQE-374 Project: Kiali QE Issue Type: QE Task Reporter: Filip Brychta Assignee: Filip Brychta Priority: Minor Labels: infrastructure Some slaves fail to start with following errors: from jenkins log: ARNING: Error in provisioning; agent=KubernetesSlave name: jenkins-slave-kiali-ui-tests-fcdsz, template=PodTemplate{inheritFrom='', name='jenkins-slave-kiali-ui-tests', namespace='jenkins-slaves', label='python kiali-ui-tests', nodeSelector='', nodeUsageMode=NORMAL, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], containers=[ContainerTemplate{name='jnlp', image='docker-registry.default.svc:5000/jenkins-slaves/jenkins-slave-kiali-ui-tests', alwaysPullImage=true, workingDir='/home/jenkins', command='', args='${computer.jnlpmac} ${computer.name} ', resourceRequestCpu='', resourceRequestMemory='', resourceLimitCpu='', resourceLimitMemory='', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@3106f760}], yaml=} java.lang.IllegalStateException: Node was deleted, computer is null at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:177) at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:292) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Aug 08, 2018 8:38:46 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate INFO: Terminating Kubernetes instance for agent jenkins-slave-kiali-ui-tests-fcdsz Aug 08, 2018 8:38:46 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate SEVERE: Computer for agent is null: jenkins-slave-kiali-ui-tests-fcdsz WARNING: Unable to move atomically, falling back to non-atomic move. java.nio.file.NoSuchFileException: /var/lib/jenkins/nodes/jenkins-slave-kiali-ui-tests-fcdsz/atomic9143280938281774341tmp -> /var/lib/jenkins/nodes/jenkins-slave-kiali-ui-tests-fcdsz/config.xml at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:191) at hudson.XmlFile.write(XmlFile.java:198) at jenkins.model.Nodes.persistNode(Nodes.java:175) at jenkins.model.Nodes.addNode(Nodes.java:144) at jenkins.model.Jenkins.addNode(Jenkins.java:2058) at hudson.slaves.NodeProvisioner$2.run(NodeProvisioner.java:241) at hudson.model.Queue._withLock(Queue.java:1380) at hudson.model.Queue.withLock(Queue.java:1257) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:207) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:61) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(No WARNING: Unable to move /var/lib/jenkins/nodes/jenkins-slave-kiali-ui-tests-fcdsz/atomic9143280938281774341tmp to /var/lib/jenkins/nodes/jenkins-slave-kiali-ui-tests-fcdsz/config.xml. Attempting to delete /var/lib/jenkins/nodes/jenkins-slave-kiali-ui-tests-fcdsz/atomic9143280938281774341tmp and abandoning. Aug 08, 2018 8:38:34 AM hudson.slaves.NodeProvisioner$2 run WARNING: Provisioned agent Kubernetes Pod Template failed to launch java.nio.file.NoSuchFileException: /var/lib/jenkins/nodes/jenkins-slave-kiali-ui-tests-fcdsz/atomic9143280938281774341tmp at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at hudson.util.AtomicFileWriter.commit(AtomicFileWriter.java:206) at hudson.XmlFile.write(XmlFile.java:198) at jenkins.model.Nodes.persistNode(Nodes.java:175) at jenkins.model.Nodes.addNode(Nodes.java:144) at jenkins.model.Jenkins.addNode(Jenkins.java:2058) at hudson.slaves.NodeProvisioner$2.run(NodeProvisioner.java:241) From docker log: INFO: Handshaking Aug 08, 2018 12:38:43 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to jenkins2.bc.jonqe.lab.eng.bos.redhat.com:39765 Aug 08, 2018 12:38:44 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP4-connect Aug 08, 2018 12:38:44 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Remote identity confirmed: b0:58:42:9c:19:76:a1:78:81:79:d9:fc:9a:e9:19:fd Aug 08, 2018 12:38:44 PM org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer onRecv INFO: [JNLP4-connect connection to jenkins2.bc.jonqe.lab.eng.bos.redhat.com/10.16.23.71:39765] Local headers refused by remote: Unknown client name: jenkins-slave-kiali-ui-tests-fcdsz Aug 08, 2018 12:38:44 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Protocol JNLP4-connect encountered an unexpected exception java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: jenkins-slave-kiali-ui-tests-fcdsz at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223) at hudson.remoting.Engine.innerRun(Engine.java:609) at hudson.remoting.Engine.run(Engine.java:469) Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Unknown client name: jenkins-slave-kiali-ui-tests-fcdsz at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.newAbortCause(ConnectionHeadersFilterLayer.java:378) at org.jenkinsci.remoting.protocol.impl.ConnectionHeadersFilterLayer.onRecvClosed(ConnectionHeadersFilterLayer.java:433) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832) at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287) at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:172) at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832) at org.jenkinsci.remoting.protocol.NetworkLayer.onRecvClosed(NetworkLayer.java:154) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$1500(BIONetworkLayer.java:48) at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:247) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) at java.lang.Thread.run(Thread.java:748) Suppressed: java.nio.channels.ClosedChannelException ... 7 more Aug 08, 2018 12:38:44 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to jenkins2.bc.jonqe.lab.eng.bos.redhat.com:39765 Aug 08, 2018 12:38:44 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Server reports protocol JNLP4-plaintext not supported, skipping Aug 08, 2018 12:38:44 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Server reports protocol JNLP3-connect not supported, skipping Aug 08, 2018 12:38:44 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Server reports protocol JNLP2-connect not supported, skipping Aug 08, 2018 12:38:44 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Server reports protocol JNLP-connect not supported, skipping Aug 08, 2018 12:38:44 PM hudson.remoting.jnlp.Main$CuiListener error SEVERE: The server rejected the connection: None of the protocols were accepted java.lang.Exception: The server rejected the connection: None of the protocols were accepted at hudson.remoting.Engine.onConnectionRejected(Engine.java:670) at hudson.remoting.Engine.innerRun(Engine.java:634) at hudson.remoting.Engine.run(Engine.java:469)