Total Order Anycast in JGroups 3.1.0.Alpha4
by Bela Ban
FYI,
I've integrated Pedro's TOA protocol into JGroups. The protocol to be
used is org.jgroups.protocols.tomTOA.
If a message has an AnycastAddress (new class) with a list of cluster
members as destination, and TOA is at the top of the stack config (see
toa.xml), then that message will be sent with total order (unless flag
NO_TOTAL_ORDER is set, then the message will be sent as a regular
anycast message using multiple unicasts).
If an RPC is sent with a destination list and
RequestOptions.use_anycast_addrs=true, then we'll also use TOA to
establish total order.
This is probably only interesting for Mircea (integrating TOA into
Infinispan) and Pedro (seeing if his changes to CommandAwareDispatcher
and the interceptor) can be simplified.
Do you guys need a release, e.g. 3.1.0.Beta1 ?
Cheers,
--
Bela Ban, JGroups lead (http://www.jgroups.org)
12 years, 7 months
Let me understand DIST
by Bela Ban
Can you confirm that my understanding of how DIST works is correct ?
#1 Non transactional modifications
- E.g. a PUT
- The PUT is sent to the primary owner (e.g. B) and all backup owners
(e.g. C) and is applied immediately, with only local lock acquisition
(lock-put-unlock)
(or is the PUT only sent to B, which in turn then updates C?)
#2 Transactional modifications
- The modifications involve a bunch of keys
- When the TX commits:
- A PREPARE message with the relevant keys is sent to all primary
owners P (to the backup owners as well?)
- All primary owners try to apply the modifications, acquiring locks
- If all primary owners can successfully apply all of the
modifications, the TX commits, else it rolls back
- After a successful TX, the primary owners update the backup owners:
here, I'm probably wrong, and this is done *inside* of the TX scope, right ?
So is my understanding of #1 and #2 is correct ?
If so, then I can assume that a transactional modification touching a
number of keys will almost always touch *all* nodes ? Example:
- We have 10 nodes
- numOwners = 2
- If we have a good consistent hash, I can assume that I have to modifiy
5 different keys (10 / 2) on average in a TX to touch *all* nodes in the
cluster with the PREPARE/COMMIT phase, correct ?
If my last statement is correct, is it safe to assume that with DIST and
transactional modifications, I will have a lot of TX contention /
collisions ?
If this is correct, this would IMO lay even more importance onto the
work done by the Cloud-TM team, replacing 2PC with total order. Also, if
we touch almost all nodes, would it make sense to use SEQUENCER for
*all* updates ? Would this obviliate the need for TOM (total order for
partial replication) ?
Well, probably not, because we only want to send keys to nodes that
actually need to store them...
Thoughts ?
--
Bela Ban, JGroups lead (http://www.jgroups.org)
12 years, 7 months
Non-blocking state transfer (ISPN-1424)
by Dan Berindei
Hi guys
It's been a long time coming, but I finally published the non-blocking
state transfer draft on the wiki:
https://community.jboss.org/wiki/Non-blockingStateTransfer
Unlike my previous state transfer design document, I think I've
fleshed out most of the implications. Still, there are some things I
don't have a clear solution for yet. As you would expect it's mostly
around merging and delayed state transfer.
I'm looking forward to hearing your comments/advice!
Cheers
Dan
PS: Let's discuss this over the mailing list only.
12 years, 7 months
TOM
by Bela Ban
Hi Pedro,
I'm currently looking at your TOM protocol (GROUP_MULTICAST).
I have a couple of points to discuss:
#1 We need to find a better package name for org.jgroups.protocols.pmcast
#2 We also need to find a better name for the protocol GROUP_MULTICAST.
In JGroups, we call a message to a single cluster member a *unicast*, a
message to all cluster members a *multicast* and a message to a subset
of the cluster members an *anycast*. So I guess both the package name
and the protocol name should have something to do with "totally ordered"
and "anycast"...
How about
- TOA (Total Order Anycast) ?
- TOPR (Total Order for Partial Replication) ?
- TOS (Total Order for a Subset) ?
-
Thoughts / suggestions ?
#3 I assume your changes to Infinispan involve sending a 1-phase
PREPARE/COMMIT message to N members (who own the keys touch by the TX).
In Infinispan, this is done via (IIRC) N unicasts, or - if this was
changed in the meantime - by an *anycast*.
Say we have to send the 1PC PREPARE message to {A,C,D,E}. So this would
be 4 unicasts (to A, C, D and E) or an anycast to the target set
{A,C,D,E}. The latter will result in 4 unicasts, like the former.
The way we could do that is to have Infinispan pass the target list
{A,C,D,E} to RpcDispatcher.callRemoteMethods() as targets. Also,
Infinispan would have to use a new option (e.g.
use_total_order_anycast). When this flag is set, in
RequestCorrelator.sendRequest(), we'd create a GroupAddress={A,C,D,E}
and send a new Message with the GroupAddress as destination down the stack.
The new protocol would then see that the Message.dest is a GroupAddress
and handle the message (not passing it further down, as the other
protocols can't handle GroupAddresses).
Is this the change you made in your prototype used in the paper
"Exploiting Total Order Multicast..." to run the benchmarks, comparing
TOM to 2PC ?
--
Bela Ban, JGroups lead (http://www.jgroups.org)
12 years, 7 months
Design of temote event handling in Hot Rod
by Galder Zamarreño
Hi all,
Re: https://community.jboss.org/docs/DOC-17571
Over the past week and a bit I've been working on a rough prototype for remote event handling in Hot Rod that covers the server side (I've not done any work on the Hot Rod client).In the link above you can find my design notes.
I wanted to get some feedback on the minimum requirements explained and I wanted to discuss the need of the optional requirements, in particular the 1st of the optional requirements.
The idea is that at a logical level, it'd be interesting to know what the origin of a modification for a couple of reasons:
- It allows clients be able to know whether the modification is originated locally from a logical point of view. If it sounds to abstract, think about near caches (see preso in https://www.jboss.org/dms/judcon/presentations/London2011/day1track2sessi...) and imagine a local cache (near cache) configured with a remote cache store (a Java hot rod client with N channels open at the same time). Remote listeners could decide to act differently if the modification was originated locally or not, i.e. if it's not local then remove the key from cache, if local, it means the modification comes from this remote cache store and so I have the latest data in memory. This is a very nice optimisation for at least this use case.
- This can be extended further. If all channels opened with can be associated with a logical origin, we could optimise sending back events. For example, imagine a remote cache store (it has 1 remote cache manager) that has N channels open with the server. There's no need for all N channels to receive a notification for a cache removal. As long as one of the channels gets it event, it's good enough to act to on the local cache.
As you can see, what I'm heading towards is that for each remote cache manager started to be id'd uniquely, and this id to be shipped with all Hot Rod operations. It could be possible to limit the operations that carry such an id, but this could complicate the protocol.
Thoughts?
Also, any thoughts on the need for the 4th optional requirement? For near caches, remote events are important, but you could limit the need for it with an aggressive eviction policy in the near cache to cover against lost events.
Cheers,
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
12 years, 7 months
Infinispan AS7 domain quickstart deployment "fun"
by Galder Zamarreño
Hi,
I've been having a bit of "fun" deploying Infinispan AS7 quickstarts in domain mode.
1. Trying to deploy Infinispan AS7 domain quickstart (https://github.com/galderz/infinispan-quickstart/tree/t_710final/jboss-as7) into AS 7.1.0.Final fails with:
https://issues.jboss.org/browse/AS7-4048
2. This appears to be fixed in AS 7.1.1.Final but in this version, there's another error when you deploy the quickstart via the CLI:
[Server:server-four] 08:53:47,370 INFO [org.jboss.as.server] (host-controller-connection-threads - 1) JBAS015870: Deploy of deployment "infinispan-jboss-as7.war" was rolled back with failure message JBAS014750: Operation handler failed to complete
I asked Emanuel about this and he's told me that this is fixed in master (??).
3. Right, so let's have a go at deploying the quickstart in master domain mode. Doesn't work:
[Server:server-three] 09:26:02,823 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC00001: Failed to start service jboss.deployment.unit."infinispan-jboss-as7.war".WeldService: org.jboss.msc.service.StartException in service jboss.deployment.unit."infinispan-jboss-as7.war".WeldService: org.jboss.weld.exceptions.DefinitionException: WELD-001450 Interceptor method public java.lang.Object org.jboss.solder.exception.control.ExceptionHandledInterceptor.passExceptionsToSolderCatch(javax.interceptor.InvocationContext) throws java.lang.Throwable does not declare that it throws Exception.
[Server:server-three] at org.jboss.as.weld.services.WeldService.start(WeldService.java:83)
[Server:server-three] at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
[Server:server-three] at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]
[Server:server-three] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [classes.jar:1.6.0_29]
[Server:server-three] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [classes.jar:1.6.0_29]
[Server:server-three] at java.lang.Thread.run(Thread.java:680) [classes.jar:1.6.0_29]
[Server:server-three] Caused by: org.jboss.weld.exceptions.DefinitionException: WELD-001450 Interceptor method public java.lang.Object org.jboss.solder.exception.control.ExceptionHandledInterceptor.passExceptionsToSolderCatch(javax.interceptor.InvocationContext) throws java.lang.Throwable does not declare that it throws Exception.
[Server:server-three] at org.jboss.weld.interceptor.reader.InterceptorMetadataUtils.isValidBusinessMethodInterceptorMethod(InterceptorMetadataUtils.java:98)
[Server:server-three] at org.jboss.weld.interceptor.reader.InterceptorMetadataUtils.isInterceptorMethod(InterceptorMetadataUtils.java:53)
[Server:server-three] at org.jboss.weld.interceptor.reader.InterceptorMetadataUtils.buildMethodMap(InterceptorMetadataUtils.java:130)
[Server:server-three] at org.jboss.weld.interceptor.reader.InterceptorMetadataUtils.readMetadataForInterceptorClass(InterceptorMetadataUtils.java:38)
[Server:server-three] at org.jboss.weld.interceptor.reader.cache.DefaultMetadataCachingReader$1.apply(DefaultMetadataCachingReader.java:30)
[Server:server-three] at org.jboss.weld.interceptor.reader.cache.DefaultMetadataCachingReader$1.apply(DefaultMetadataCachingReader.java:28)
[Server:server-three] at com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference.compute(ComputingConcurrentHashMap.java:355)
[Server:server-three] at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.compute(ComputingConcurrentHashMap.java:184)
[Server:server-three] at com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.getOrCompute(ComputingConcurrentHashMap.java:153)
[Server:server-three] at com.google.common.collect.ComputingConcurrentHashMap.getOrCompute(ComputingConcurrentHashMap.java:69)
[Server:server-three] at com.google.common.collect.ComputingConcurrentHashMap$ComputingMapAdapter.get(ComputingConcurrentHashMap.java:393)
[Server:server-three] at org.jboss.weld.interceptor.reader.cache.DefaultMetadataCachingReader.getInterceptorMetadata(DefaultMetadataCachingReader.java:54)
[Server:server-three] at org.jboss.weld.bean.InterceptorImpl.<init>(InterceptorImpl.java:64)
[Server:server-three] at org.jboss.weld.bean.InterceptorImpl.of(InterceptorImpl.java:59)
[Server:server-three] at org.jboss.weld.bootstrap.AbstractBeanDeployer.createInterceptor(AbstractBeanDeployer.java:228)
[Server:server-three] at org.jboss.weld.bootstrap.BeanDeployer.createBeans(BeanDeployer.java:137)
[Server:server-three] at org.jboss.weld.bootstrap.BeanDeployment.createBeans(BeanDeployment.java:204)
[Server:server-three] at org.jboss.weld.bootstrap.WeldBootstrap.deployBeans(WeldBootstrap.java:349)
[Server:server-three] at org.jboss.as.weld.WeldContainer.start(WeldContainer.java:82)
[Server:server-three] at org.jboss.as.weld.services.WeldService.start(WeldService.java:76)
[Server:server-three] ... 5 more
Kevin/Pete, any idea what's up this time?
Please, let's try to get this fixed before 7.1.2.Final is out so that we can finally have the Infinispan quickstarts working again.
Cheers,
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
12 years, 7 months
Semaphore vs Lock
by Galder Zamarreño
Hi,
I was reading up about Java's Semaphores (http://docs.oracle.com/javase/6/docs/api/java/util/concurrent/Semaphore.html) and a couple of ideas came to my mind:
1. Wouldn't it make sense to use binary semaphores instead of locks in Infinispan? We're already having to override ReentrantLock in order to have locks owned by Transactions rather than threads. Initially I thought it might make easier for deadlock detection, but not so sure right now cos we're already changing things to avoid thread ownership of locks.
2. Could lock striping become lock pooling with a simple object pool based on a Semaphore? In theory, we'd avoid the current issue with lock striping where two diff locks hash to the same segment and we have deadlocks. We could use, as the current lock striping logic does, the concurrency level to decide the number of semaphore permits.
I'll prototype this on the side and see if I can make such conversion and see if it brings any benefits… :)
In the mean time, if you have any feedback to provide, it'd be appreciated.
Cheers,
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
12 years, 7 months
5.1.3.CR1
by Galder Zamarreño
Hi all,
We're doing an Infinispan 5.1.3.CR1 tomorrow.
If you have any issues pending for this version, could you please look into them before then?
Cheers,
--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache
12 years, 7 months