[Design of Messaging on JBoss (Messaging/JBoss)] - Re: Server side HA and failover
by clebert.suconic@jboss.com
I have *tried* to drive fail over from client side.
Last week I could make a state transfer from one node to another, moving all the messages from a failed queue to a new queue.
There are implications of doing this way as we will could have more than one Local Queues when a failover occurs.
Say if you have two nodes A and B, each node with a client and a durable subscription on that client.
Now say if node A fails. Now the clients from node A will be redirect into node B, and Routers will have to treat B's connections as they were still on nodeA but now they are local on node B.
Also any producer existent on NodeA will come to nodeB having preferences to send messages to the former NodeA.
These are all implications of the Partial Queue implementation existent in our cluster. I'm at this point looking at the code on how we could deal with this.
--Clebert
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3980866#3980866
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3980866
19 years, 5 months
[Design of Messaging on JBoss (Messaging/JBoss)] - Re: Client failover redeliveries discussion
by clebert.suconic@jboss.com
Ovidiu wrote :
| Let's systematize a bit the cases we need to deal with on fail-over.
|
| ......
|
| Again, for all these cases we should have tests.
|
I havent thought much about retries and stuff like that. I agree we should create test for these scenarios.
At this point I'm concentrating in transfer state from one connection to another in conjunction with server side failover.
I will start treating connection events problems after I have finished this state transfer.
Ovidiu wrote :
| This will help us avoid situation when a "failed-over" message is consumed on the client-side, the consumer delegate sends the acknowledgment and the new server consumer endpoint doesn't know what the client is talking about, because there's no active delivery for that message. I think we can go a step further and also send the IDs of non-persistent messages that have been "failed-over" on the client side. This way, the client will continue to receive (and successfully acknowledge) non-persistent messages that otherwise would have been lost.
|
Let me make some progress with the server side failover, then we will decide what to do with these IDs. We should do it next week.
Ovidiu wrote :
| We need to make sure that the "new" connection is then properly discarded so it will eventually garbage collected. Minor thing, anyway, just an observation.
|
agreed.
Ovidiu wrote :
| Clebert wrote :
| | Creating a new connection on the new server will create a new server Object, consequently a new ServerId.
| |
|
| You probably want to say that creating a new connection will subsequently create a new server-side connection endpoint instance, and because we're connecting to a different server node, we need to use its serverID instead of the "dead node" serverID, right?
|
Yep... I have recently tried to improved my terminology. That was sent before your request ;-)
Ovidiu wrote :
| What about the "fail-over protocol"? Your statement above seem to assume that the new server node is called into without any "preparation", as would a completely new client that creates a new connection, session and consumer endpoints. This is not going to work, those server-side objects need to undergo a "post-failover" preparation phase, where deliveries for the client-side failed over messages are created and so forth.
|
I'm working on the preparation/failover protocol right now. I'm trying to drive it from the client as the first connection fails to a node appears.
"Ovidiu" wrote :
| At this point, given the way Remoting discussion evolve, we can safely assume we'll use Remoting connection failure facilities to detect our connection failure.
|
"Ovidiu" wrote :
| I have started a list of scenarios at the beginning of this post. The way should be deal with this situation is to create test cases for each of use cases we are going to support. This way we make sure that if we refactor the code later, it still behaves how we originally wanted it to. No new functionality should ideally be added if there is no test case for it.
|
Yep!
I will let Tim comment on the list.
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3980863#3980863
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3980863
19 years, 5 months