[infinispan-issues] [JBoss JIRA] (ISPN-5315) Tight loop on ClientListenerNotifier

Tight loop on ClientListenerNotifier ------------------------------------ Key: ISPN-5315 URL: https://issues.jboss.org/browse/ISPN-5315 Project: Infinispan Issue Type: Bug Components: Listeners Affects Versions: 7.2.0.Beta1, 7.1.1.Final Reporter: Gustavo Fernandes Assignee: Gustavo Fernandes Priority: Minor Fix For: 7.2.0.Beta2, 7.2.0.Final Attachments: stackTrace Sometimes the {{ClientListenerNotifier}} keeps printing extremely high number of warning and quickly fill GBs of disk space: {code} 11:28:16,805 WARN [ClientListenerNotifier] (Client-Listener-aacb322ffc60405e) ISPN004039: Unable to complete reading event from server null java.nio.channels.IllegalBlockingModeException at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:201) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:179) at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:282) at org.infinispan.client.hotrod^C.impl.protocol.Codec20.readEvent(Codec20.java:126) at org.infinispan.client.hotrod.event.ClientListenerNotifier$EventDispatcher.run(ClientListenerNotifier.java:237) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) {code} This can be reproduced when running the server test suite, more specifically the test {{ExampleConfigsIT}}. Right after the server is started the above warning is logged in a loop for the duration of the tests. I've started to observe this after commit (1), which changed the behaviour to retry on SocketTimeouts. Before the aforementioned commit, log were printing: {code} 10:49:56,361 ERROR [ClientListenerNotifier] (Client-Listener-b53963d5ab2b4234) ISPN004043: Unrecoverable error reading event from server null, exiting event reader thread org.infinispan.client.hotrod.exceptions.TransportException:: java.net.SocketTimeoutException at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:184) at org.infinispan.client.hotrod.impl.protocol.Codec20.readMagic(Codec20.java:282) at org.infinispan.client.hotrod.impl.protocol.Codec20.readEvent(Codec20.java:126) at org.infinispan.client.hotrod.event.ClientListenerNotifier$EventDispatcher.run(ClientListenerNotifier.java:236) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.SocketTimeoutException at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.readByte(TcpTransport.java:179) ... 8 more {code} And thus the listener thread was exiting, (1) https://github.com/infinispan/infinispan/commit/1a03a0a84307ae8b95c06b398...