[Design of JBossCache] - Re: Locking tree before setting initial state
by bstansberry＠jboss.com
Continuing the recreation of the e-mail conversation...
"bela(a)jboss.com" wrote :
| anonymous wrote :
| | #2. Won't any PREPARE received on the state provider side after the state transfer request just be waiting in the channel for the getState() to complete? However, these kinds of PREPARE calls are presumably being handled on the state transfer *recipient* side as well, and can thus lead to inconsistent state. So, having the *recipient* return FAIL from PREPARE is needed.
|
| True. So, if the state transfer takes longer than the PREPARE timeout, the initiator will return with a timeout exception and rollback the TX anyway. So we don't need to do anything here. If the state transfer takes less, the PREPAREs will succeeed *after* the state transfer has completed.
|
| anonymous wrote : #3A5 -- Breaking locks for crashed members is triggered by receipt of a view change. But what if the view change is stuck in the channel behind the blocked getState() call? Again, we may need a "priority queue" concept to deal with this. This is tricky though, because here the "priority queue" is in JGroups, and basically means letting view change messages jump ahead of others. I certainly wouldn't think that could be the default behavior in JGroups, and even if it was a configurable option I imagine it could cause problems even in the JBC use case.
|
| I don't feel very good about view changes jumping ahead of regular messages, this could badly screw up the QoS provided by JGroups. For
| example: Virtual Synchrony mandates that all messages sent in view V1 are *delivered* in V1 *before* V2 is installed (this is done through the FLUSH protocol), so having V2 jump ahead of regular messages violates this property.
|
| anonymous wrote : Actually, once FLUSH is in place, if the getState() call discovers a lock held by a remote gtx, the existence of the lock is actually proof of a dead (or at least very unhealthy) member, isn't it?
|
| Yes, absolutely ! The FLUSH will probably have to have a timeout associated with it, all locks still held after this phase can be broken.
|
| anonymous wrote : #3B1. Even with FLUSH, it is possible the getState() call won't be able to acquire a lock. This would be because a tx initiated on the state provider itself is holding the lock and has not yet started the 2PC phase. This kind of tx can safely be rolled back though. Other kinds of long lasting locally-initiated locks are possible too, though unlikely. E.g. a simple non-transactional put where the calling thread got hung up in a badly written TreeCacheListener. So, the lock-breaking code is still partly useful.
|
| *We *MUST* absolutely change this behavior and throw an exception if the state cannot be retrieved !!!*
|
| anonymous wrote :
| | This is a problem that has existed in JBC all along. I don't see a good fix for it in 1.3. Adding a lot of changed behaviors that cause rolled back transactions but which don't really ensure a valid state transfer seems like making trouble. Maybe better to just leave the behavior from 1.2.4 and wait for the real fix in 1.4.
|
| Yes, I agree. With the change, of course, that we fail if the state
| cannot be retrieved. Maybe we should also increase the state transfer
| timeout in the XML default files
|
|
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3970500#3970500
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3970500
19 years, 7 months
[Design of JBossCache] - Re: Locking tree before setting initial state
by bstansberry＠jboss.com
Continuing the recreation of the e-mail conversation...
"bstansberry(a)jboss.com" wrote :
| Some comments on your analysis:
|
| #1A. The way it is coded now is that if a lock is held by a non-local gtx that is STATUS_PREPARED, the lock is just broken; the tx is not affected. This is because as you noted once a non-local tx is STATUS_PREPARED it is too late to notify the intiator. So, the lock is just broken. This basically means the state provider is sending uncommitted state. If the gtx later commits, no harm, but if the initiator of the gtx later issues a rollback(), the state transfer recipient will have invalid state. Doing it this way is less likely to lead to inconsistent state than rolling back the tx on the state provider, but it is still not good.
|
| #1B. The PREPARE locks won't ever be released (at least without adding something like a "priority queue" to JBC -- JBCACHE-326). The COMMIT message will be stuck in the JGroups channel behind the blocked getState() call.
|
| #2. Won't any PREPARE received on the state provider side after the state transfer request just be waiting in the channel for the getState() to complete? However, these kinds of PREPARE calls are presumably being handled on the state transfer *recipient* side as well, and can thus lead to inconsistent state. So, having the *recipient* return FAIL from PREPARE is needed.
|
| #3A5 -- Breaking locks for crashed members is triggered by receipt of a view change. But what if the view change is stuck in the channel behind the blocked getState() call? Again, we may need a "priority queue" concept to deal with this. This is tricky though, because here the "priority queue" is in JGroups, and basically means letting view change messages jump ahead of others. I certainly wouldn't think that could be the default behavior in JGroups, and even if it was a configurable option I imagine it could cause problems even in the JBC use case.
|
| Actually, once FLUSH is in place, if the getState() call discovers a lock held by a remote gtx, the existence of the lock is actually proof of a dead (or at least very unhealthy) member, isn't it?
|
| #3B1. Even with FLUSH, it is possible the getState() call won't be able to acquire a lock. This would be because a tx initiated on the state provider itself is holding the lock and has not yet started the 2PC phase. This kind of tx can safely be rolled back though. Other kinds of long lasting locally-initiated locks are possible too, though unlikely. E.g. a simple non-transactional put where the calling thread got hung up in a badly written TreeCacheListener. So, the lock-breaking code is still partly useful.
|
|
| The killer issue is #1B. Until we do FLUSH or JBCACHE-326, the possibility exists that getState() will get stuck. Seems like in that situation we have a choice -- let the getState() call fail, or break the locks and accept the risk of inconsistent state. If we let the getState() call fail, then we have the choice of what to do on the recipient side -- abort the startup of the cache, or let the cache start without transferred state. Up to 1.3.0, JBC has been written to allow getState() to fail and let the recipient cache start without the transferred state.
|
| My current (not very strong) inclination Bela is to agree with your daughter when she wrote last week that "The proper and *only* way IMO is to wait until FLUSH has been implemented in JGroups 2.4"
| This is a problem that has existed in JBC all along. I don't see a good fix for it in 1.3. Adding a lot of changed behaviors that cause rolled back transactions but which don't really ensure a valid state transfer seems like making trouble. Maybe better to just leave the behavior from 1.2.4 and wait for the real fix in 1.4.
|
| One thing I definitely think is that if we are going to allow the getState() call to fail, we shouldn't have the recipient return FAIL to prepare() calls while it waits for the state. That's just a nightmare -- rolled back transactions all over the place, and then the new cache ends up without a state transfer anyway :(
|
|
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3970499#3970499
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3970499
19 years, 7 months
[Design of JBossCache] - Re: Locking tree before setting initial state
by bstansberry＠jboss.com
Following the previous post last March, a rather extensive discussion took place on private e-mail. 6 months later I'm going to recreate the highlights of that discussion here so the whole thread will be available in one place as we discuss these issues again next week:
"bela(a)jboss.com" wrote : I thought about this, and I think the best thing is to *not* break the locks ! Let's look at the use cases.
|
| #1 PREPARE was received and now state transfer request is received (state provider side)
|
| We cannot break the locks acquired by PREPARE because we returned OK from PREPARE, so the coordinator assumes the TX will be successful (assuming for now everyone returned OK). Returning OK from PREPARE means we will be able to apply the changes, so we cannot break the locks acquired by PREPARE ! Otherwise, the result might be that everyone applied a TX, except the state provider, so we end up with inconsistent state...
|
| Looks like we have to wait for the PREPARE locks to be released. This shouldn't take long though
|
|
| #2 PREPARE is received after state transfer request has been received (again, on state provider side)
|
| We cannot queue the PREPAREs, for similar reasons as discussed in #1: returning OK from a PREPARE means we successfully acquired the locks, but if we queue, we don't know whether we will be able to acquire the locks later !
|
| We can throw an exception so we return FAIL from PREPARE, which means that during state transfer, nobody will be able to commit transactions across the cluster !
|
| #3 PREPARE and FLUSH
|
| Once we have FLUSH, we do the following:
|
| a) On state transfer, we run the FLUSH protocol
| b) It flushes pending multicasts out of the system so that all members have received the same messages after a FLUSH
| c) It blocks members from sending new messages, this is done via the block() callback in MembershipListener
| d) The block() needs to do the following:
| --- Stop sending new messages (in ReplicationInterceptor)
| --- Complete existing PREPARE rounds, e.g. return from block() only when a 2PC has been committed or rolled back
| e) This means that when the state provider receives a state transfer request, the entire tree will be *lock-free*, because all 2PCs have completed. The only time this isn't true is when the 2PC leader crashed so we have dangling locks. However, Brian fixed this recently.
|
| So what do we need to do once we have FLUSH and the block() callback is implemented correctly ?
| We simply revert back to the original state transfer code, which acquires all locks on the state provider side (with small timeouts), and we don't need to break any locks !
|
|
| So the strategy is to throw exceptions to PREPARE in JBossCache 1.3.0 (#2b), and in 1.4.0 we use solution #3 (depends on JGroups 2.4 with FLUSH).
|
|
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3970498#3970498
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3970498
19 years, 7 months
[Design of EJB 3.0] - Re: Specifying Default Client Binding
by ALRubinger
Perhaps this isn't a bug, but IMHO makes developing very difficult.
>From ProxyDeployer.initializeRemoteBindingMetadata():
if (binding == null)
| {
| ...
| }
| else
| {
| RemoteBinding[] list = {binding};
| remoteBindings = new RemoteBindingsImpl(list);
| advisor.getAnnotations().addClassAnnotation(RemoteBindings.class, remoteBindings);
| }
Saying - if you've specified @RemoteBinding, use the bind parameters it contains. One of these is clientBindUrl, which gets a default value of socket://0.0.0.0:3873 unless explicitly overridden in the annotation.
What if you've specified in the Service Bindings manager that your EJBs should be contacted via an alternate port? The only way to contact on the new port is to specify the client bind URL in the annotation? I'd like to be able to deploy my EJBs into any configured server instance without telling it where it should be listening; that's already been accomplished for all my other components in the Service Bind Manager.
Unfortunately as it stands it's impossible to determine whether the @RemoteBinding.clientBindUrl value was set through the default, or specified by the user. I'd recommend having the default value of @RemoteBinding.clientBindUrl be null, and changing the code in ProxyDeployer to the following:
else
| {
| // If the @RemoteBinding.clientBindUrl was specified
| // by the developer, use it, otherwise use the default
| RemoteBinding[] list = {new RemoteBindingImpl(
| binding.jndiBinding(),
| binding.interceptorStack(),
| (binding.clientBindUrl() == null) ? this.defaultClientBinding : binding.clientBindUrl(),
| binding.factory())};
| remoteBindings = new RemoteBindingsImpl(list);
| advisor.getAnnotations().addClassAnnotation(RemoteBindings.class, remoteBindings);
| }
Thoughts?
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3970474#3970474
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3970474
19 years, 7 months
[Design of Messaging on JBoss (Messaging/JBoss)] - JBMESSAGING-541 - Multiple connections on a single thread wi
by clebert.suconic＠jboss.com
I"ve opened this thread to discuss the creationg of DurableCrashTest1 and DurableCrashTest2.
If I have a client with two connections opened (don't know if the fact they are in the same thread matters or not), the lease timer stops and never comes back.
I've discovered this by adding a couple of log.info into JBossRemoting during leasing and stop leasing. And basically, during addCallBAck a new client is created/closed what stops the lease timer.
What is funny is with a single connection test, this works fine. (I mean the connection is detected without any problem)
I have also opened a JIRA on JBossRemoting:
http://jira.jboss.org/jira/browse/JBREM-596
public void testSimplestDurableSubscription() throws Exception
| {
| ConnectionFactory cf = (ConnectionFactory)ic.lookup("ConnectionFactory");
| Topic topic = (Topic)ic.lookup("/topic/TopicCrash");
|
| Connection conn = cf.createConnection();
| conn.setClientID("client1");
| conn.start();
|
| Connection conn2 = cf.createConnection(); // funny... if I remove everything related to conn2, the lease thread works
| conn2.setClientID("client2");
| conn2.start();
|
| Session s = conn.createSession(true, Session.AUTO_ACKNOWLEDGE);
| Session s2 = conn2.createSession(true, Session.AUTO_ACKNOWLEDGE);
|
| MessageProducer prod = s.createProducer(topic);
| prod.setDeliveryMode(DeliveryMode.PERSISTENT);
|
| MessageConsumer durable = s.createDurableSubscriber(topic, "subs1");
| MessageConsumer durable2 = s2.createDurableSubscriber(topic, "subs2");
|
| //conn2.close(); -- do not remote this comment. It was meant to not close the connection
|
|
| for (int i=0;i<10;i++)
| {
| prod.send(s.createTextMessage("k"+i));
| }
| s.commit();
|
| conn.close();
| conn = cf.createConnection();
| conn.setClientID("client1");
| conn.start();
| s = conn.createSession(true, Session.AUTO_ACKNOWLEDGE);
| durable = s.createDurableSubscriber(topic, "subs1");
|
| for (int i=0;i<2;i++)
| {
| TextMessage tm = (TextMessage)durable.receive(1000);
| assertNotNull(tm);
| s.commit();
| System.out.println(tm.getText());
| assertEquals("k" + i, tm.getText());
| }
|
| //conn.close();-- do not remote this comment. It was meant to not close the connection
|
| //System.exit(0); -- this is not needed as there is not tearDown, the client VM will simply be finished the same way an exit would do, and this is better since it will keep proper JUNIT report outputs
| }
|
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3970472#3970472
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3970472
19 years, 7 months