[Design of Messaging on JBoss (Messaging/JBoss)] - Re: JBoss 2.0 Thread model
by timfox
dmlloyd said:
anonymous wrote :
| "jmesnil" wrote : All JMS connections and their children (session, producer, consumer) to the same JMS server use a single MINA NioSocketConnector (i.e. one single TCP socket).
|
| This is very similar to what I'm doing with Remoting 3 ("R3"), and I've run into the same issues as you.
|
| In R3, the protocol consists of message datagrams tunneled over a single TCP connection. We have the additional issue that certain messages must be well-ordered with respect to certain other messages (though this isn't always the case; some messages can be processed in any order). In order to maximize throughput, we want to process messages in parallel whenever it's OK to do so.
|
| To achieve this, I've done two things. First, every incoming message is assigned to an Executor. Second, I've got a simple class called OrderedExecutorFactory which can produce Executor instances which execute in order with respect to that Executor, but in parallel with respect to other ordered Executors and unordered tasks. (see http://tinyurl.com/33b4gq)
|
| As an aside, by using Executors rather than a ThreadFactory, I can push the responsibility of maintaining a thread pool off to someone else; yet a simple implementation is still available in the standalone case by doing Executors.newCachedThreadPool() for example.
|
| Since the ordering of messages is a detail of the protocol, the messages have to be at least partially decoded before they can be assigned to an Executor. Therefore I don't bother with IoFilters or anything like that - I've found that it's simpler to just have an IoHandler and do all the message decoding and Executor delegation within the handler.
|
| I also hope to add support for SCTP in the future, which allows multiple independent streams within a single connection (among other cool features), reducing this "head-of-line" contention issue. However, while I've heard that there is an implementation in the works at Sun, it's still probably a ways off yet (unless one wants to use APR, which adds other dependencies as well).
|
| One thing I was thinking about as a possible performance enhancement would be to have more than one connection - not one per session necessarily, but maybe like 2-4 connections, using some type of load-balancing among them. This should give SCTP-like behavior (well, to a limited extent) with reduced head-of-line contention, but without a ridiculous explosion of connections.
|
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4140635#4140635
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4140635
16 years, 2 months
[Design of Messaging on JBoss (Messaging/JBoss)] - JBoss 2.0 Thread model
by timfox
OOOPS! Sorry I accidentally deleted this thread while doing some forum housekeeping! :)
Here it is in an ugly form I'm afraid:
jmesnil said:
anonymous wrote :
| JBM 2 is using MINA as its remoting API.
| All JMS connections and their children (session, producer, consumer) to the same JMS server use a single MINA NioSocketConnector (i.e. one single TCP socket).
| This means that all the JMS resources associated to the same JMS server share the same MINA IoSession.
| In turn, this means that the messages sent by 2 JMS Sessions created from the same JMS Connection are ordered globally. This is a performance-killer and less than optimal : order must be ensured at the JMS Session level only.
|
| One potential solution is to use 1 MINA IoSession per JMS Session + 1 MINA IoSession for each JMS connection.
| This solution is simple to implement.
| However this means having many TCP sockets open between a JMS client and a server...
|
| The other solution is to introduce a customized ExecutorFilter to process MINA messages in other threads than the I/O process thread and still ensure that messages associated to a JMS Session and its children are treated by the same thread.
|
| How to implement this ExecutorFilter?
|
| The filter will have a ThreadFactory.
| When a MINA message is sent/received, we look for a thread associated to the message target (AbstractPacket.targetID).
| We need to correlate the session ID based on this targetID (the target can be a JMS Producer or Consumer)
| Once we got the JMS Session ID, we use it a key to look up in a Map to get a Thread in return.
| If there is such a thread, we use it.
| Otherwise, we get a new thread and associate it to the JMS Session ID.
| We can use a WeakValueHashmap (from jboss-common) with the targetID as the key and the thread as the weak value.
|
| Thread life cycle
|
| We get a new thread when we see a new JMS Session ID.
| However, we do not know a priori when all messages targeted to a JMS Session have been send or received.
| We can parse all the messages to find one corresponding to closing the JMS Session but is not a good idea to have to parse every message to identify this few messages.
| Instead, the threads can have a keep alive time (e.g. 60 seconds) before being reclaimed (and thus being removed from the WeakValueHashmap).
| If a thread associated to a JMS Session ID is reclaimed while the Session is still open, we create a new one next time we receive a message with this ID.
|
| We also need to have threads to execute JMS Connection messages.
|
| ID correlation
|
| For now, each JMS resources has a random UUID.
| We need to be able to deduce a JMS Session ID from one of its children ID (Producer, Consumer, QueueBrowser).
| Their IDs could be prepended by a JMS Session ID (e.g. <session ID>/<resource ID>).
| Or we can add a new attribute to the Packet interface (e.g. resourceID) which will be set either to a JMS Connection ID (for JMS Connections) or a JMS Session ID (for JMS Sessions, Producers, Consumers & QueueBrowsers).
| This resourceID will be used as the key to the WeakValueHashMap.
| This latest solution avoid parsing text for every message but adds a bit to the message size.
|
| wdyt?
|
View the original post : http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4140634#4140634
Reply to the post : http://www.jboss.com/index.html?module=bb&op=posting&mode=reply&p=4140634
16 years, 2 months