[Design of Messaging on JBoss (Messaging/JBoss)] - Re: Messaging and Remoting
by tom.elrod＠jboss.com
1. Messaging needs a working multiplexed transport.
There is currently a multiplex transport (as Ovidiu noted) and how it works under the hood is explained at (http://labs.jboss.com/portal/jbossremoting/docs/multiplex/multiplex.html). Its performance is not great (see http://labs.jboss.com/portal/jbossremoting/docs/benchmark/performance_ben... for comparison benchmarks). Ron can probably
I have been looking at Mladen's MUX project as another option for a multiplex transport to remoting. However, it does not automatically handle calling back over the same physical connection, so this behavior will need to either be added within the MUX project or on top of it. I have not done much testing with the MUX project other than some simple coding to see how it works/behaves. No idea on performance, but know that Mladen says he still wants some more time to work on it before he feel comfortable doing a GA release for it.
2. Remoting uses a server-side blocking thread-per-connection model.
So guess would be good to cover scenarios and best approaches for them. The first scenario would be having a limited set (in the low hundreds) of clients that each regularly make invocations. In this scenario, having optimal throughput is probably the desired behavior and the current remoting code is designed for this. At the other end of the spectrum would be having a large number of clients (in the thousands) that each make periodic requests (meaning is not in tight loop that continually makes invocations). The best scenario for this is to use non-blocking i/o. The main reason for this is if use blocking i/o (besides running out of resources) will have thread per request and will be slow as balls because of all the thread context switching for all the concurrent requests. Then there are the scenarios in between (btw can add config so that remoting does not hold connection after invocation on the server side so that will free worker threads faster, which I am pretty sure is already in jira).
As Ovidiu said, I am looking into adding NIO (and APR) currently. However, I don't want anyone thinking that will see a performance gain from the NIO transport until reach a certain threshold of clients (probably somewhere between 300-700, but don't have anything concrete on this yet). Of course, there are still a ton a factors that influence this as well such as number of cpus, jvm, network, etc.
3. Remoting uses a client-side blocking request-response invocation model.
Have made the code changes (for jira issue http://jira.jboss.com/jira/browse/JBREM-548) so that if doing one-way invocations, only the request ever gets written to the wire and that thread on the client returns immediately after the write. On the server side, it never writes the response back to client.
Probably good to have better definition of what asynchronous means. There are a lot of places the term can be used. At a high level in the call stack, can just mean using a future model to make call and wait for a response to come back at later point in time (or just a callback on a different thread at later point in time). At lowest level, could be talking about only writing on the wire and not waiting for response off the wire. Lots of places term can be used in between the high and low level.
4. Messaging needs the ability to send unformatted data on the wire.
It is possible to send and receive raw payload data using remoting where is not wrapped in the remoting InvocationRequest/InvocationResponse using configuration per request. However, the behavior of doing this is not consistent over all the current transports. Also, sending raw data will prevent a lot of the extra remoting features from being used as internal remoting metadata is included in the InvocationRequest/InvocationResponse wrappers.
Another issue is the types of streams being given to the marshalling layer being inconsistent across the different transports (this is covered in http://jira.jboss.com/jira/browse/JBREM-597). My preference is to only send the raw OutputStream/InputStream to the marshalling layer and allow the Marshaller/UnMarshaller implementations decide on what type of streams to wrap around the raw ones passed in (if any).
There is certainly a lot of re-working that can be done in this general area and hoping the efforts for NIO transport will help bring these out.
As for AMQP, I don't now anything about it, so can?t comment.
As for remoting and its purpose, it is exactly as Juha stated which is to provide a remoting platform that other JEMS projects can use (but imo should be by choice). I have tried my best to do this in a way that any new feature or implementation does not negatively impact any other project that may be using remoting. The only jira issue derived from the messaging project that has been rejected is http://jira.jboss.com/jira/browse/JBREM-371, which I couldn't do per previous sentence. It is my guess that at least a third of the development effort for remoting this year has been related to supporting the messaging project (which has all been good for the remoting project in general). The remoting team is myself and Ron Sigal (who is now splitting development time between remoting and messaging). There are 51 remaining open issues for the next remoting release (plus many external ones related to remoting).
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3978948#3978948
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3978948
19 years, 5 months
[Design of Messaging on JBoss (Messaging/JBoss)] - Re: Client failover redeliveries discussion
by ovidiu.feodorov＠jboss.com
Clebert wrote :
| Basically what I do on failover is to create a new consumer on the new server, replace IDs and re-register the Callback handler. The server at this point will think it's a new client coming. In case of durable subscribers/queues and persistent messages you will have the queue refilled
|
Could you explain in a little bit more detail how this works? You detect the failure at remoting level, and then, what happens? Do you preserve the Connection/Session/Consumer hierarchy on the client? How do you create the corresponding endpoint hierarchy on the fail-over ServerPeer?
(btw, for the sake of clarity, when you describe a process, could you please qualify the actors a little bit better. For example, instead of saying "what I do on failover is to create a new consumer", you probably wanted to say "what I do on failover is to create a new ClientConsumerDelegate (?) instance")
Clebert wrote :
| If the consumer receives a message from CallBack but if it didn't send an ACK yet, after the failover, the server not knowing the message might throw an exception (messageId not known).
|
The client stack knows that the fail-over took place, so if the message is non-persistent, it doesn't need to send the ACK anyway (from the server's perspective, the message is lost), and if the message is persistent, it will be recovered on the fail-over server, so the ACK will arrive for a known message. Am I am missing something?
Tim wrote :
| Yes - we should send the ids of every persistent message as part of the failover protocol - the server then repopulates the delivery list in the server consumer endpoint
|
What do you mean send the ids of every persistent message as part of the failover protocol. Who sends the ids? The messages are in database, until they are ACKed, and this is where the fail-over server will recover them from, and repopulate the queue.
Clebert wrote :
| I'm considering having a conference call with developers about these possibilities.
|
Before having a conference call, I think a better idea is to summarize what you've implemented so far. Describe how the fail-over happens step by step. Expand http://wiki.jboss.org/wiki/Wiki.jsp?page=NewMessagingHADesign for that.
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3978935#3978935
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3978935
19 years, 5 months
[Design of Messaging on JBoss (Messaging/JBoss)] - Re: Messaging and Remoting
by bill.burke＠jboss.com
"bill.burke(a)jboss.com" wrote : anonymous wrote :
| | The ServerInvoker has a bounded "ServerThread"s thread pool, the number of simultaneous invocations that can be handled at a time is limited by invoker's maxPoolSize configuration parameter. Once a "ServerThread" is allocated to a client connection, it will stay associated to that connection until the client closes the socket or a socket timeout occurs.This could lead to thread starvation in presence of a large number of clients. Increasing the number of server threads will draw down the performance because the large number of context switches.
| |
|
| IIRC, Tom took a lot from the PooledInvoker. The PooledInvoker had a LRU queue that closed idle connections when one was needed so that there was no starving. Also, the PooledInvoker tried to avoid a large number of context switches by associating the client connection to a specific server thread. Therefore, no thread context switches were required. Of course, this design was predicated on "keep alive" clients. So if the use case was very "keep alive" oriented, it performed really well.
|
| For the TransactionManager recover prototype I did, I experimented with a batch queue for writing to a log. Obviously having one thread exclusively write to the file was a lot faster and scalable than individual threads writing to the one file. It would be cool to combine both the concepts of pooling and this type of multiplexing. Never did the benchmarks on that type of design.
|
|
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3978924#3978924
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3978924
19 years, 5 months
[Design of Messaging on JBoss (Messaging/JBoss)] - Re: Messaging and Remoting
by bill.burke＠jboss.com
anonymous wrote :
| The ServerInvoker has a bounded "ServerThread"s thread pool, the number of simultaneous invocations that can be handled at a time is limited by invoker's maxPoolSize configuration parameter. Once a "ServerThread" is allocated to a client connection, it will stay associated to that connection until the client closes the socket or a socket timeout occurs.This could lead to thread starvation in presence of a large number of clients. Increasing the number of server threads will draw down the performance because the large number of context switches.
|
Basically, my thoughts on multiplexing are that it is good for certain usecases bad for others. For instance, many remote clients do not have more than one thread talking to the server. While a web application might have many threads talking to one remote JMS server (HA JMS?) At what point does it make sense to multiplex on the client side? What point does it make sense to multiplex on the server side?
The largest overhead and performance drain is in creating connections. IIRC, Tom took a lot from the PooledInvoker design which I wrote. The PooledInvoker was written in mind of a long running client that needs to continuously talk to the server and avoid creating new connections.
I think you should make a few prototypes and bench. Maybe the Tomcat guys have a lot of insight into this?
Bill
A non-blocking NIO model would probably more appropriate here. Other messaging projects use this approach already. I know that Tom is working on adding NIO support to Remoting, and maybe this is all we need. It would be nice if can share his thoughts on the subject here.
3. Remoting uses a client-side blocking request-response invocation model.
Remoting is essentially a mechanism through which a client sends an invocation over the network and blocks waiting the response. This is not always what Messaging needs. Messaging clients should be able to send data on the wire without the unnecessarily added burden of waiting for a reply. Similarly, the server should be able to asynchronously send data to the client.
Remoting simulates this situation by using a client-side worker thread that blocks instead of the calling thread, and throwing away the server response. A variation of this mechanism uses a server-side pooled asynchronous processing threads that throws away the response on the server. The common theme here is "throwing away" the reply.
We are wondering what would it take to amend Remoting API (and implementation) to support pure asynchronous client calls.
4. Messaging needs the ability to send unformatted data on the wire.
This is required to support AMQP. Remoting wraps everything in Invocation Request instances.
Personally I think that adding the functionality we need (inside or outside of Remoting), instead of replacing Remoting altogether is the most efficient way to go forward. If we go for a new implementation, we need to add by ourselves functionality that comes for free in Remoting, such as:
* Connection failure notification and leasing/pinging
* Automatic local invocation optimization (pass by reference)
* Pluggable transports, specifically HTTP(S) and SSL.
* Versioning support (however, this actually may prove to be a problem, in what AMQP support is concerned).
Opinions?
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3978919#3978919
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3978919
19 years, 5 months
[Design of Messaging on JBoss (Messaging/JBoss)] - Re: Messaging and Remoting
by juha＠jboss.org
"ovidiu.feodorov(a)jboss.com" wrote : In practical terms, we need to decide if is feasible to extend JBoss Remoting in a way that helps Messaging to reach its functionality and performance targets, or replace it with a specialized Messaging transport layer, independent of JBoss Remoting.
|
The latter should be a non-option. Remoting was created especially to be reused across JEMS so that there's a unified approach to "remoting". If it doesn't fit in the current requirements of JEMS projects it needs to be modified to do so. If this is not done, then I would question the whole point of the project.
Now whether or not all the points you listed can be incorporated in Remoting in due time for Messaging's deliverables schedule is another question. If not then implement the features you need in quick and dirty fashion, and put down blocker task requests for all those features in the next release of Remoting. Whatever you do outside Remoting should be considered throw-away code (if not reusable inside Remoting).
anonymous wrote :
| Personally I think that adding the functionality we need (inside or outside of Remoting), instead of replacing Remoting altogether is the most efficient way to go forward.
|
In my opinion the only question should be where all the feature requirements you list are going to fit in Remoting's release schedule and write them down as JIRA tasks. Remoting filling those feature requests must be the final goal. If not in the current work-in-progress release, then the next one after that.
My cents deposited.
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3978916#3978916
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3978916
19 years, 5 months
[Design of Messaging on JBoss (Messaging/JBoss)] - Messaging and Remoting
by ovidiu.feodorov＠jboss.com
This is yet another discussion thread on the subject of Messaging/Remoting integration.
The previous one (http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3971228) ended nowhere. What is different about the current one is that I will manage it until we reach conclusions on all points in contention. We need to take a decision fast, since the remoting problem is a dependency of everything else in 1.2.Beta. In practical terms, we need to decide if is feasible to extend JBoss Remoting in a way that helps Messaging to reach its functionality and performance targets, or replace it with a specialized Messaging transport layer, independent of JBoss Remoting.
This is the list of points of contention:
1. Messaging needs a working multiplexed transport.
This is documented by the JIRA issue http://jira.jboss.org/jira/browse/JBMESSAGING-92. We want to use the same physical TCP/IP connection to send data from client to server AND asynchronously, from server to client. Once this is available, the client won't need to create its physical callback server anymore. Callback servers cannot be used in firewalled environments, and even in common LAN environments they are liable to introduce complications.
Support for multiplexed invokers exists in JBoss Remoting (http://labs.jboss.com/portal/jbossremoting/docs/guide/ch10.html#d0e5713). Tim worked on integrating it to a certain point, and then he stopped on account of performance issues, if I remember correctly. It would be interesting if he shares his conclusions here, so all of us understand what didn't work.
2. Remoting uses a server-side blocking thread-per-connection model.
The ServerInvoker has a bounded "ServerThread"s thread pool, the number of simultaneous invocations that can be handled at a time is limited by invoker's maxPoolSize configuration parameter. Once a "ServerThread" is allocated to a client connection, it will stay associated to that connection until the client closes the socket or a socket timeout occurs.This could lead to thread starvation in presence of a large number of clients. Increasing the number of server threads will draw down the performance because the large number of context switches.
A non-blocking NIO model would probably more appropriate here. Other messaging projects use this approach already. I know that Tom is working on adding NIO support to Remoting, and maybe this is all we need. It would be nice if can share his thoughts on the subject here.
3. Remoting uses a client-side blocking request-response invocation model.
Remoting is essentially a mechanism through which a client sends an invocation over the network and blocks waiting the response. This is not always what Messaging needs. Messaging clients should be able to send data on the wire without the unnecessarily added burden of waiting for a reply. Similarly, the server should be able to asynchronously send data to the client.
Remoting simulates this situation by using a client-side worker thread that blocks instead of the calling thread, and throwing away the server response. A variation of this mechanism uses a server-side pooled asynchronous processing threads that throws away the response on the server. The common theme here is "throwing away" the reply.
We are wondering what would it take to amend Remoting API (and implementation) to support pure asynchronous client calls.
4. Messaging needs the ability to send unformatted data on the wire.
This is required to support AMQP. Remoting wraps everything in Invocation Request instances.
Personally I think that adding the functionality we need (inside or outside of Remoting), instead of replacing Remoting altogether is the most efficient way to go forward. If we go for a new implementation, we need to add by ourselves functionality that comes for free in Remoting, such as:
* Connection failure notification and leasing/pinging
* Automatic local invocation optimization (pass by reference)
* Pluggable transports, specifically HTTP(S) and SSL.
* Versioning support (however, this actually may prove to be a problem, in what AMQP support is concerned).
Opinions?
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=3978910#3978910
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=3978910
19 years, 5 months