[infinispan-issues] [JBoss JIRA] (ISPRK-22) InfinispanRDD is not fault tolerant

[Vojtech Juranek (JIRA)]

Vojtech Juranek created ISPRK-22:
------------------------------------

             Summary: InfinispanRDD is not fault tolerant
                 Key: ISPRK-22
                 URL: https://issues.jboss.org/browse/ISPRK-22
             Project: Infinispan Spark
          Issue Type: Bug
          Components: RDD
    Affects Versions: 0.3
            Reporter: Vojtech Juranek


When primary ISPN server fails during processing InfinispanRDD, Spark is not able to overcome this failure. 
This is caused by re-creating {{RemoteCachManager}} with pre-configured ISPN server address (for read [here|https://github.com/infinispan/infinispan-spark/blob/master/src/main/scala/org/infinispan/spark/rdd/InfinispanRDD.scala#L66], for writes [here|https://github.com/infinispan/infinispan-spark/blob/master/src/main/scala/org/infinispan/spark/package.scala#L33]), so when this server fails during RDD processing and Spark calls some function, which under the hood creates {{RemoteCacheManager}}, it will fail with connection refused exception.

[Here|https://github.com/vjuranek/infinispan-spark/commit/9aef6b7269feadb9c8543485e2beec51bb667447] are some basic tests (requires some more work, but should be fine for reproducing the issue) and example of exception thrown by HR client:
{noformat}
org.infinispan.client.hotrod.exceptions.TransportException:: Could not fetch transport
        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:395)
        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:241)
        at org.infinispan.client.hotrod.impl.operations.FaultTolerantPingOperation.getTransport(FaultTolerantPingOperation.java:26)
        at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:53)
        at org.infinispan.client.hotrod.impl.RemoteCacheImpl.ping(RemoteCacheImpl.java:490)
        at org.infinispan.client.hotrod.impl.RemoteCacheImpl.resolveCompatibility(RemoteCacheImpl.java:551)
        at org.infinispan.client.hotrod.RemoteCacheManager.createRemoteCache(RemoteCacheManager.java:341)
        at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:222)
        at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:217)
        at org.infinispan.spark.rdd.InfinispanRDD$$anonfun$1.apply(InfinispanRDD.scala:52)
        at org.infinispan.spark.rdd.InfinispanRDD$$anonfun$1.apply(InfinispanRDD.scala:52)
        at scala.Option.map(Option.scala:146)
        at org.infinispan.spark.rdd.InfinispanRDD.compute(InfinispanRDD.scala:52)
        at org.infinispan.spark.rdd.InfinispanRDD.compute(InfinispanRDD.scala:66)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
        at org.apache.spark.scheduler.Task.run(Task.scala:89)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: /127.0.0.1:11222
        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:78)
        at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:35)
        at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:16)
        at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:390)
        ... 21 more
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111)
        at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:68)
        ... 25 more
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)