[infinispan-issues] [JBoss JIRA] (ISPRK-22) InfinispanRDD is not fault tolerant

[Gustavo Fernandes (JIRA)]

     [ https://issues.jboss.org/browse/ISPRK-22?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gustavo Fernandes updated ISPRK-22:
-----------------------------------
    Status: Open  (was: New)


> InfinispanRDD is not fault tolerant
> -----------------------------------
>
>                 Key: ISPRK-22
>                 URL: https://issues.jboss.org/browse/ISPRK-22
>             Project: Infinispan Spark
>          Issue Type: Bug
>          Components: RDD
>    Affects Versions: 0.3
>            Reporter: Vojtech Juranek
>            Assignee: Gustavo Fernandes
>
> When primary ISPN server fails during processing InfinispanRDD, Spark is not able to overcome this failure. 
> This is caused by re-creating {{RemoteCachManager}} with pre-configured ISPN server address (for read [here|https://github.com/infinispan/infinispan-spark/blob/master/src/main/scala/org/infinispan/spark/rdd/InfinispanRDD.scala#L66], for writes [here|https://github.com/infinispan/infinispan-spark/blob/master/src/main/scala/org/infinispan/spark/package.scala#L33]), so when this server fails during RDD processing and Spark calls some function, which under the hood creates {{RemoteCacheManager}}, it will fail with connection refused exception.
> [Here|https://github.com/vjuranek/infinispan-spark/commit/c4a3101624e76d61097c4f27157f9e815e63d57b] are some basic tests and example of exception thrown by HR client:
> {noformat}
> org.infinispan.client.hotrod.exceptions.TransportException:: Could not fetch transport
>         at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:395)
>         at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.getTransport(TcpTransportFactory.java:241)
>         at org.infinispan.client.hotrod.impl.operations.FaultTolerantPingOperation.getTransport(FaultTolerantPingOperation.java:26)
>         at org.infinispan.client.hotrod.impl.operations.RetryOnFailureOperation.execute(RetryOnFailureOperation.java:53)
>         at org.infinispan.client.hotrod.impl.RemoteCacheImpl.ping(RemoteCacheImpl.java:490)
>         at org.infinispan.client.hotrod.impl.RemoteCacheImpl.resolveCompatibility(RemoteCacheImpl.java:551)
>         at org.infinispan.client.hotrod.RemoteCacheManager.createRemoteCache(RemoteCacheManager.java:341)
>         at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:222)
>         at org.infinispan.client.hotrod.RemoteCacheManager.getCache(RemoteCacheManager.java:217)
>         at org.infinispan.spark.rdd.InfinispanRDD$$anonfun$1.apply(InfinispanRDD.scala:52)
>         at org.infinispan.spark.rdd.InfinispanRDD$$anonfun$1.apply(InfinispanRDD.scala:52)
>         at scala.Option.map(Option.scala:146)
>         at org.infinispan.spark.rdd.InfinispanRDD.compute(InfinispanRDD.scala:52)
>         at org.infinispan.spark.rdd.InfinispanRDD.compute(InfinispanRDD.scala:66)
>         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>         at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
>         at org.apache.spark.scheduler.Task.run(Task.scala:89)
>         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: org.infinispan.client.hotrod.exceptions.TransportException:: Could not connect to server: /127.0.0.1:11222
>         at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:78)
>         at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:35)
>         at org.infinispan.client.hotrod.impl.transport.tcp.TransportObjectFactory.makeObject(TransportObjectFactory.java:16)
>         at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
>         at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransportFactory.borrowTransportFromPool(TcpTransportFactory.java:390)
>         ... 21 more
> Caused by: java.net.ConnectException: Connection refused
>         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>         at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
>         at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111)
>         at org.infinispan.client.hotrod.impl.transport.tcp.TcpTransport.<init>(TcpTransport.java:68)
>         ... 25 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.11#64026)