Asynchronous cache's "void put()" call expectations changed from 6.0.0 to 6.0.1/7.0
by Galder Zamarreño
Hi all,
@Paul, this might be important for WF if using async repl caches (the same I think applies to distributed async caches too)
Today I’ve been trying to upgrade Infinispan version in Hibernate master from 6.0.0.Final to 7.0.0.Beta1. Overall, it’s all worked fine but there’s one test that has started failing.
Essentialy, this is a clustered test for a repl async cache (w/ cluster cache loader) where a non-owner cache node does put() and immediately, on the same cache, it calls a get(). The test is failing because the get() does not see the effects of the put(), even if both operations are called on the same cache instance.
According to Dan, this should have been happening since [1] was implemented, but it’s really started happening since [2] when lock delegation was enabled for replicated caches (EntryWrappingInterceptor.isUsingLockDelegation is now true whereas in 6.0.0 it was false).
Not sure we set expectations in this regard, but clearly it’s big change in terms of expectations on when “void put()” completes for async repl caches. I’m not sure how we should handle this, but it definitely needs some discussion and adjuts documentation/javadoc if needed. Can we do something differently?
Indepent of how we resolve this, this is the result of once again of trying to shoehole async behaviour into sync APIs. Any async caches (DIST, INV, REPL) should really be accessed exclusively via the AsyncCache API, where you can return quickly and use the future, and any listener to attach to it (a bit ala Java8’s CompletableFuture.map lambda calls) as a way to signal that the operation has completed, and then you have an API and cache mode that make sense and is consistent with how async APIs work.
Right now, when a repl async cache’s “void put()” call is not very well defined. Does it return when message has been put on the network? What impact does it have in the local cache contents?
Also, a very big problem of the change of behaviour is that if left like that, you are forcing users to code differently, using the same “void put()” API depending on the configuration (whether async/sync). As clearly shown by the issue above, this is very confusing. It’s a lot more logical IMO, and I’ve already sent an email on this very same topic [3] back in January, that whether a cache is sync or async should be based purely on the API used and forget about the static configuration flag on whether the cache is async or sync.
Cheers,
[1] https://issues.jboss.org/browse/ISPN-2772
[2] https://issues.jboss.org/browse/ISPN-3354
[3] http://lists.jboss.org/pipermail/infinispan-dev/2014-January/014448.html
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
9 years, 1 month
Infinispan tutorial
by Tristan Tarrant
Hi guys,
I've been working on how to spruce up our website, docs and code samples.
While quickstarts are ok, they come as monolithic blobs which tell you
nothing about how you got there. For this reason I believe a
step-by-step tutorial approach is better and I've been looking at the
AngularJS tutorials [0] as good examples on how to achieve this.
I have created a repo [1] on my GitHub user where each commit is a step
in the tutorial. I have tagged the commits using 'step-n' so that you
can checkout any of the steps and run them:
git checkout step-1
mvn clean package exec:java
The GitHub web interface can be used to show the diff between steps, so
that it can be linked from the docs [2].
Currently I'm not aiming to build a real application (although
suggestions are welcome in this sense), but just going through the
basics, adding features one by one, etc.
Comments are welcome.
Tristan
---
[0] https://docs.angularjs.org/tutorial/step_00
[1] https://github.com/tristantarrant/infinispan-embedded-tutorial
[2]
https://github.com/tristantarrant/infinispan-embedded-tutorial/compare/st...
9 years, 3 months
Fix or document? Concurrent replaceWithVersion w/ same value might all return true - ISPN-4972
by Galder Zamarreño
Hi all,
Re: https://issues.jboss.org/browse/ISPN-4972
Embedded cache provides atomicity of a replace() call passing in the previous value. This limitation might be lifted when we adopt Java 8 and we can pass in a lambda or similar, which can be executed right when the value is compared now, and if it returns true it’s applied. The lambda could compare both value and metadata for example.
Anyway, given the current status, I’m considering whether it’s worth fixing this particular issue. Fixing the issue would require adding some kind of locking in the Hot Rod server so that the version retrieval, comparison and replace call, can all happen atomically.
This is not ideal, and on top of that, as Radim said, the chances of this happening in real life are limited, or more precisely it’s effects are minimal. In other words, if two concurrent threads call replace with the same value, the end result is that the new value would be stored, but as a result of the code, both replaces would return true which is not strictly right.
I’d rather document this than add unnecessary locking in the Hot Rod server where it deals with the versioned replace call.
Thoughts?
--
Galder Zamarreño
galder(a)redhat.com
twitter.com/galderz
9 years, 4 months
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
by Andreas Kruthoff
Hi infinispan-dev
I'm running 2 processes with 2 distributed caches each, standard
jgroups-tcp configutation. Both caches have a local dat file which is
loaded during startup, passivation is true. Each cache contains ~20Mio.
entries.
I'm writing with async put, peak is over 10'000 entries per second. It
performs well.
cache.getAdvancedCache()
.withFlags(Flag.SKIP_REMOTE_LOOKUP, Flag.SKIP_CACHE_LOAD)
.putIfAbsentAsync(Long.valueOf(a), Long.valueOf(b));
As soon as I launch a 3rd process to join the 2 caches, I'm getting the
following exception (see below).
Does anyone know what I need to tune. It looks like the OS doesn't offer
enough resources, or am I wrong? The server has plenty of RAM and CPU's.
I'm launching without -Xmx, but with -XX:+UseG1GC.
Any help is much appreciated
-andreas
Exception in thread "main"
org.infinispan.manager.EmbeddedCacheManagerStartupException:
org.infinispan.commons.CacheException: Unable to invoke method public
void org.infinispan.remoting.transport.jgroups.JGroupsTransport.start()
on object of type JGroupsTransport
at
org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:243)
at
org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:573)
at
org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:539)
at
org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:416)
at ch.nexustelecom.lbd.engine.ImsiCache.init(ImsiCache.java:49)
at
ch.nexustelecom.dexclient.engine.DefaultDexClientEngine.init(DefaultDexClientEngine.java:120)
at
ch.nexustelecom.dexclient.DexClient.initClient(DexClient.java:169)
at
ch.nexustelecom.dexclient.tool.DexClientManager.startup(DexClientManager.java:196)
at
ch.nexustelecom.dexclient.tool.DexClientManager.main(DexClientManager.java:83)
Caused by: org.infinispan.commons.CacheException: Unable to invoke
method public void
org.infinispan.remoting.transport.jgroups.JGroupsTransport.start() on
object of type JGroupsTransport
at
org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:170)
at
org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:869)
at
org.infinispan.factories.AbstractComponentRegistry.invokeStartMethods(AbstractComponentRegistry.java:638)
at
org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:627)
at
org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:530)
at
org.infinispan.factories.GlobalComponentRegistry.start(GlobalComponentRegistry.java:221)
... 8 more
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at
org.jgroups.protocols.FD_SOCK$ServerSocketHandler.start(FD_SOCK.java:1006)
at
org.jgroups.protocols.FD_SOCK$ServerSocketHandler.<init>(FD_SOCK.java:999)
at org.jgroups.protocols.FD_SOCK.init(FD_SOCK.java:188)
at
org.jgroups.stack.ProtocolStack.initProtocolStack(ProtocolStack.java:860)
at org.jgroups.stack.ProtocolStack.setup(ProtocolStack.java:481)
at org.jgroups.JChannel.init(JChannel.java:848)
at org.jgroups.JChannel.<init>(JChannel.java:159)
at org.jgroups.JChannel.<init>(JChannel.java:129)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.buildChannel(JGroupsTransport.java:381)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.initChannel(JGroupsTransport.java:286)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.initChannelAndRPCDispatcher(JGroupsTransport.java:330)
at
org.infinispan.remoting.transport.jgroups.JGroupsTransport.start(JGroupsTransport.java:189)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
... 13 more
This email and any attachment may contain confidential information which is intended for use only by the addressee(s) named above. If you received this email by mistake, please notify the sender immediately, and delete the email from your system. You are prohibited from copying, disseminating or otherwise using the email or any attachment.
9 years, 4 months
Failover Implementation with RANDOM_NODE_FAILOVER not working
by rtom
I'm trying to implement a basic failover policy for a cluster of 2 nodes.
The task that I want to run is a DistributedCallable object and I create a
DistributedTask for it. Based on the output files, the task is being run
correctly on the cluster (I would sometimes see it run on server 1 and other
times on server 2 and it completes).
I decided to go with the random node failover policy that is provided and
when the task is running and I kill the server that is running the task, I
don't see the other server picking up the task and running it.
I'm not too sure if I'm missing anything when I'm creating and executing the
distributedtask:
DistributedTaskBuilder<Boolean> taskBuilder =
execService.createDistributedTaskBuilder(usageReportingProcess);
taskBuilder =
taskBuilder.failoverPolicy(DefaultExecutorService.RANDOM_NODE_FAILOVER);
DistributedTask<Boolean> distTask = taskBuilder.build();
Future<Boolean> future = execService.submit(distTask);
Any insight or tips would be very helpful
Thanks!
--
View this message in context: http://infinispan-developer-list.980875.n3.nabble.com/Failover-Implementa...
Sent from the Infinispan Developer List mailing list archive at Nabble.com.
9 years, 4 months
Functionality based on configuration
by Radim Vansa
Hi,
when thinking about strong/eventual consistency and ease of
configuration, I was considering whether cache configuration should
affect results of operations at all (one example could be read
committed/repeatable read, or write skew check).
It would seem to me that the configuration would be simpler, and user
options more rich if those options that change the result of operation
would be purely API-wise (based on flags or method arguments) and the
configuration could only change the performance (defining cache store
will slow down some operations) or availability of these operations (you
cannot start a transaction when the manager is not defined), not the
outcome.
E.g. is there really a point to be able to change sync/async
configuration of the cache when the code expects strong consistency? If
it can handle that, it should grab cache.withFlags(FORCE_ASYNCHRONOUS)
and work on that.
Another example is in the strong/eventual consistency - if I want to see
the cache as strongly consistent, I can't read from backup owners [1].
Currently there is no option to force reading from primary owner,
therefore, I was wondering whether it should be configurable (together
with staggered gets policy - not that this would be implemented) or
whether that should be specified as a flag - and it seems to me that it
should not be configurable as the administrator could remove the flag
from the config (and see increased performance) but eventually a race
could occur where this flag matters and the application will behave
incorrectly.
WDYT? This question is obviously rather for changes on the roadmap (I'd
say along with leaving ConcurrentMap interface) than any immediate
actions in versions 7.x or 8.x.
Radim
[1] https://issues.jboss.org/browse/ISPN-4995
--
Radim Vansa <rvansa(a)redhat.com>
JBoss DataGrid QA
9 years, 4 months
AsyncCacheWriter is dead
by Andreas Kruthoff
Hi dev
I'm running infinispan-7.0.1 (will soon upgrade to 7.0.2).
I've configured 2 distributed caches, both are similar.
Example cache1:
...
<distributed-cache name="infinicache-lbd-imei" mode="SYNC"
async-marshalling="false" owners="2"
l1-lifespan="600000" l1-cleanup-interval="60000" statistics="false">
<eviction strategy="LIRS" max-entries="20000000"/>
<expiration max-idle="86400000" interval="30000"/>
<persistence passivation="true">
<file-store
path="/data1/infini-lbd"
preload="true" shared="true" purge="false">
<write-behind/>
</file-store>
</persistence>
</distributed-cache>
...
Today, I've seen that one of the two caches suddenly doesn't exist
anymore somehow (JMX access to numberOfEntries returns nothing?!).
After looking into the application logfile, I've found this:
2014-11-19 13:45:47,310 ERROR
[AsyncStoreCoordinator-infinicache-lbd-imei] AsyncCacheWriter.java:267
ISPN000055: Unexpected error in AsyncStoreCoordinator thread.
AsyncCacheWriter is dead!
org.infinispan.util.concurrent.TimeoutException: ISPN000233: Waiting on
work threads latch failed:
java.util.concurrent.CountDownLatch@6e30928d[Count = 1]
at
org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreCoordinator.workerThreadsAwait(AsyncCacheWriter.java:297)
~[infinispan-embedded.jar:7.0.1.Final]
at
org.infinispan.persistence.async.AsyncCacheWriter$AsyncStoreCoordinator.run(AsyncCacheWriter.java:254)
~[infinispan-embedded.jar:7.0.1.Final]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_72]
No error before nor after this entry. It looks like it's related to
<write-behind/> to me, so I'll remove that setting.
But I'd like to write asynchronously for best performance.
Any help would be appreciated!
-andreas
This email and any attachment may contain confidential information which is intended for use only by the addressee(s) named above. If you received this email by mistake, please notify the sender immediately, and delete the email from your system. You are prohibited from copying, disseminating or otherwise using the email or any attachment.
9 years, 4 months
