[JBoss JIRA] Created: (JBREM-706) In socket transport, prevent client side oneway invocations from artificially reducing concurrency
by Ron Sigal (JIRA)
In socket transport, prevent client side oneway invocations from artificially reducing concurrency
--------------------------------------------------------------------------------------------------
Key: JBREM-706
URL: http://jira.jboss.com/jira/browse/JBREM-706
Project: JBoss Remoting
Issue Type: Task
Security Level: Public (Everyone can see)
Affects Versions: 2.2.0.Beta2 (Bluto)
Reporter: Ron Sigal
Assigned To: Ron Sigal
Fix For: 2.2.0.Beta2 (Bluto)
There is a subtle problem with the handling of client side oneway invocations in the socket transports (and descendants). After the invocation has been marshalled to the wire, the connection is returned to the connection pool, from which it can be reused, even if the server side handler is still processing the previous invocation. Consequently, invocations can back up and run serially instead of running concurrently.
The following dialogue exposes the problem and possible solutions:
===============================================================================================================
Ron Sigal wrote
Hi Tom,
I just noticed an interesting phenomenon that I wanted to tell you about. Maybe it'll be obvious to you, but it surprised me.
I'm writing unit tests for socket and http, since they behave differently on oneway invocations. I've got a oneway ThreadPool with two threads and I make 4 oneway invocations, one right after the other. The handler waits for 5 seconds before returning. I expected that they would all run on the server simultaneously, but they don't. What happens is something like this: (1) invocation 1 gets made on thread 1 and socket connection 1 to ServerThread 1 is created; (2) invocation 2 gets made on thread 2 and socket connection 2 to ServerThread 2 gets created; (3) invocation 3 gets made on thead 1 and socket connection 1 gets reused. But ServerThread 1 is still sleeping, so invocation 3 waits for 5 seconds.
Not a big deal, but I might not have guessed it if I hadn't seen it.
===============================================================================================================
Ron Sigal wrote
Hi Tom,
Well, it's stranger than I thought. It turns out that for client side oneway invocations, socket, which returns immediately after marshaling the invocation, can be slower than http, which waits for a response.
Consider a client side oneway threadpool with 2 threads and a queue of size 1, and suppose there are 4 invocations in quick succession. The handler takes 5 seconds per invocation.
http:
1. 1st invocation runs in the first threadpool thread.
2. 2nd invocation runs in the second threadpool thread
3. 3rd invocation goes on the queue
4 4th invocation runs on the main thread
Assuming there are at least 3 server threads, invocations 1, 2, and 4 run in the first 5 seconds, and then invocation 3 runs. Total time 10 seconds.
socket (one possible scenario):
1. 1st invocation runs in the first threadpool thread and returns the socket connection to the pool
2. 2nd invocation runs in a threadpool thread, finds the pooled connection, marshals the invocation, and returns the connection to the pool. On the server side, the ServerThread runs the invocations after finishing the 1st invocation.
3. 3rd invocation runs in a threadpool thread, finds the pooled connection, marshals the invocation, and returns the connection to the pool. On the server side, the ServerThread runs the invocations after finishing the 2nd invocation.
4. 4th invocation runs in a threadpool thread, finds the pooled connection, marshals the invocation, and returns the connection to the pool. On the server side, the ServerThread runs the invocations after finishing the 3rd invocation.
Total time: 20 seconds.
I've seen this happen. The scary thing is that the story scales. With more threads on the client side and server side, http can run more and more invocations in parallel. Whereas socket could end up running arbitrarily many invocations serially.
I think that having MicroSocketClientInvoker return after marshaling a client side oneway invocation isn't a good idea. Ovidiu's JBREM-685 ( <http://jira.jboss.com/jira/browse/JBREM-685>A server needs redundant information to detect a one way invocation) <http://jira.jboss.com/jira/browse/JBREM-685> gives me a half baked idea. Maybe these two ways of tagging a oneway invocation could be used to distinguish client side from server side oneway invocations. Or something like that.
By the way, I think that there's no problem with server side oneway invocations, since a single ServerInvoker can keep shoving invocations into the threadpool.
-Ron
===============================================================================================================
Tom Elrod wrote:
Wow. Never thought about this happening but makes sense. So the real problem with the socket client invoker is it pools the connections when doing oneway invocations. So it is queuing up invocations on the same connection which can only be processed serially on the server side. Think maybe your idea about using the ONEWAY_FLAG on the client can work. Basically am thinking we'd have to check that before returning the connection to pool and if is set, then throw away the connection (i.e. don't put back in pool). This way, the next oneway invocation would not be using the same connection and on the server side, should use another available server thread to do the process (so could be in parallel). However, this means that incur more overhead on the client side to create a new connection for each oneway invocation, but is maybe worth the cost (too bad we can't tell if serve will take a long time or not).
Let me know what you think.
===============================================================================================================
Ron Sigal wrote:
It seems to me that MicroSocketClientInvoker could:
...
2. in the case of client side oneway invocations, either throw away the connection and return immediately, or just not return until it gets a response from the server. Since it's on a separate thread, maybe the latter is preferable, since then it can return the connection to the pool after it gets a response.
===============================================================================================================
Tom Elrod wrote:
Yeah, this is a tricky one. On the one hand, if don't throw away the connection and wait for the server response AND the server takes a long time, would be possible to exhaust the pool since all threads will be stuck waiting for server to respond. With the new change to the thread pool to use the RUN mode, would mean calling threads would then be making the remote call and would pretty much be negating the purpose of oneway client side (since client caller thread would be making the network call and blocking until gets response).
However, if do throw away the connection, then means that in general have defeated purpose of connection pooling since will have to create a new connection each time oneway client side invocations are made. So if the server processes the requests quickly, would be taking on the extra overhead of creating a new client connection for every call.
Hmmm. Could go either way if feel strongly one way or another. However think I am leaning towards throwing away the connection.
===============================================================================================================
Ron Sigal wrote:
This might be complicating things, but for client side oneway invocations we could set a short per invocation timeout, say 1 second or 5 seconds. If the response comes back, put the connection back in the pool. If it times out, throw the connection away. Don't know how to pick the timeout, though.
===============================================================================================================
Tom Elrod wrote:
I hate getting too complicated as tends to increase the chance of introducing bugs. However, if wanted to do this, would say need to make the timeout very low, i.e. 1 second.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
17 years
[JBoss JIRA] Created: (GPD-138) error dialogue appears on task panel of properties for task node
by Ross McDonald (JIRA)
error dialogue appears on task panel of properties for task node
----------------------------------------------------------------
Key: GPD-138
URL: http://jira.jboss.com/jira/browse/GPD-138
Project: JBoss jBPM GPD
Issue Type: Bug
Components: jpdl
Affects Versions: jBPM JPDL Designer 3.1.0.beta1
Environment: os x macbok pro, eclipse 3.3.0
Reporter: Ross McDonald
Assigned To: Koen Aers
Priority: Minor
Hi,
when working with the designer in eclipse, if I put a new task node on the screen, immediately enter a value against the name on the 'general' panel under properties, then click on the tasks panel under properties, I get an error dialogue with the following details :
Reason:
0
'An error has occurred. See error log for more details.
0
On the other hand, if after entering a name, I click on the description input field, but do not enter anything, then click on the task panel under properties, there is no error.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
17 years
[JBoss JIRA] Created: (JBREM-843) MicroSocketClientInvoker can miscount number of active sockets
by Ron Sigal (JIRA)
MicroSocketClientInvoker can miscount number of active sockets
--------------------------------------------------------------
Key: JBREM-843
URL: http://jira.jboss.com/jira/browse/JBREM-843
Project: JBoss Remoting
Issue Type: Bug
Security Level: Public (Everyone can see)
Reporter: Ron Sigal
Assigned To: Ron Sigal
Fix For: 2.2.2.GA_CP01, 2.4.0.Beta1 (Pinto), 2.2.2.SP2
org.jboss.remoting.transport.socket.MicroSocketClientInvoker maintains (1) a pool of available sockets, and (2) a counter, usedPooled, of the number of sockets currently in use. Because the sequence of (1) checking for a free socket in the pool, and (2) creating a new socket if the pool is empty, is not in a synchronized block, it is possible to create more sockets than the maximum pool size. In particular, the following sequence of events can produce "extra" sockets:
1. Thread 1 sees that the pool is empty,
2. Thread 2 returns a socket to the pool, and
3. Thread 1 creates a new socket.
It would be possible to synchronize events 1 and 3, but it's probably undesirable since the process of getting a socket from the pool can be time consuming if the "socket.check_connection" flag is set to true. Therefore, MicroSocketClientInvoker tolerates the possibility of creating "extra" sockets. However, it never puts more than the configured maximum number of sockets in the pool. That is, if the pool is full (i.e., size == maxPoolSize), then, when an invocation completes, the socket used for that invocation will be discarded instead of being added to the pool.
The problem is that the usedPooled counter is incremented every time a socket is created and decremented every time a socket is returned to the pool, but is is *not* decremented when a socket is discarded. Suppose maxPoolSize is 100, If 101 sockets are created, 100 sockets are returned to the pool, and 1 socket is discarded, then the resulting value of usedPooled will be 1 instead of 0. It appears, then, that there is 1 socket in use, so MicroSocketClientInvoker will create no more than 99 additional sockets (excluding the race described above). So, there is a leak, not of sockets, but of "slots" for sockets. Eventually, it could become impossible to create new sockets.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
17 years
[JBoss JIRA] Created: (GPD-191) ESB Service Node
by Koen Aers (JIRA)
ESB Service Node
----------------
Key: GPD-191
URL: http://jira.jboss.com/jira/browse/GPD-191
Project: JBoss jBPM GPD
Issue Type: Feature Request
Components: jpdl
Reporter: Koen Aers
Assigned To: Koen Aers
Fix For: jBPM jPDL Designer 3.1.2
ESB Service Node
A new icon on the tool palette that would inject the following JPDL
syntax into my process definition:
<node name="Calculate Discount">
<action name="esbAction"
class="org.jboss.soa.esb.services.jbpm.actionhandlers.EsbActionHandler">
<esbCategoryName>BPM_Orchestration4</esbCategoryName>
<esbServiceName>DiscountService</esbServiceName>
<jbpmToEsbVars>
<mapping jbpm="entireCustomerAsObject" esb="customer" />
<mapping jbpm="entireOrderAsObject" esb="orderHeader" />
<mapping jbpm="entireOrderAsXML" esb="BODY_CONTENT" />
</jbpmToEsbVars>
<returnVars>
<mapping jbpm="entireOrderAsObject" esb="order"
/>
</returnVars>
</action>
<transition name="" to="Review Discount"></transition>
</node>
And provide a property sheet that allows me to specify the service name,
category and the mapping of process variables to esb message parts.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
17 years
[JBoss JIRA] Created: (JBRULES-1308) getFactHandle with equality-based assert behavior
by Adam Lewandowski (JIRA)
getFactHandle with equality-based assert behavior
-------------------------------------------------
Key: JBRULES-1308
URL: http://jira.jboss.com/jira/browse/JBRULES-1308
Project: JBoss Drools
Issue Type: Feature Request
Security Level: Public (Everyone can see)
Affects Versions: 4.0.3
Reporter: Adam Lewandowski
If I set my rule base to use equality-based assert behavior, I'm expecting to be able to use the getFactHandle(object) method to retrieve a previously asserted fact by providing an equivalent (by equals() and hashCode()) object as an argument. This appears to not be the case, as AbstractWorkingMemory uses it's identityMap and associated IdentityAssertMapComparator to do the getFactHandle lookup instead of the assertMap and EqualityAssertMapComparator. This causes the lookup to be done with the system hashCode() method and not the overridden version supplied in my fact class. The lookup thus fails and returns null.
See attached unit test.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
17 years
[JBoss JIRA] Created: (JBAS-4682) Web session passivation does very little with FIELD granularity
by Brian Stansberry (JIRA)
Web session passivation does very little with FIELD granularity
---------------------------------------------------------------
Key: JBAS-4682
URL: http://jira.jboss.com/jira/browse/JBAS-4682
Project: JBoss Application Server
Issue Type: Bug
Security Level: Public (Everyone can see)
Components: Clustering, Web (Tomcat) service
Affects Versions: JBossAS-5.0.0.Beta2
Reporter: Brian Stansberry
Assigned To: Brian Stansberry
Passivation of clustered HttpSessions works by evicting the node tree representing a given session from the cache. Configuration is based on defining how many active (in-memory) sesssion are allowed and how old they can be before passivating them. Problem is, with FIELD replication, the bulk of the data associated with a session is not in that sessions's subtree; it's in the _JBossInternal_ area. Evicting the session's tree doesn't free up the memory associated with the POJOs.
Structure of data in JBC:
/JSESSION
+++ /localhost
++++++/webapp
+++++++++/session12345
+++++++++/session67890
+++++++++/_JBossInternal_
++++++++++++/pojoAAAAA
++++++++++++/pojoBBBBB
(Note: session passivation is not done using the regular JBC eviction algorithms, as JBC doesn't understand what a "session" is. The http session replication layer manages the passivation, just using the JBC evict() method to do the actual passivation.)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
17 years