[ http://jira.jboss.com/jira/browse/JGRP-527?page=comments#action_12364808 ] Bryce Alcock commented on JGRP-527: ----------------------------------- I made some more interesting discoveries: Basically I loaded up the Multiplexer.java with some System.outs, (here are my changes) and here is the result. Note that this just repeats forever. (Well I actaully did let it run overnight just to see, and sure enough, the next morning it was still spitting out the same messages...) /** * Fetches the map of services and hosts from the coordinator (Multiplexer). No-op if we are the coordinator */ public void fetchServiceInformation() throws Exception { while(true) { Address coord=getCoordinator(), local_address=channel != null? channel.getLocalAddress() : null; boolean is_coord=coord != null && local_address != null && local_address.equals(coord); if(is_coord) { if(log.isTraceEnabled()) log.trace("I'm coordinator, will not fetch service state information"); break; } ServiceInfo si=new ServiceInfo(ServiceInfo.STATE_REQ, null, null, null); MuxHeader hdr=new MuxHeader(si); Message state_req=new Message(coord, null, null); state_req.putHeader(NAME, hdr); service_state_promise.reset(); System.out.println("state_req.toString():" +state_req.toString()); channel.send(state_req); System.out.println("channel.toString():" +channel.toString()); JChannel chan = (JChannel) channel; System.out.println("protocolSpec"+chan.printProtocolSpec(true)); System.out.println("chan.toString(treu) True....."+chan.toString(true)); System.out.println("chan.toString"+chan.toString()); try { System.out.println("service_state_promise is of type" + service_state_promise.getClass().getName()); System.out.println("service_state_promise toString()" + service_state_promise.toString()); byte[] state=(byte[])service_state_promise.getResultWithTimeout(8000); if(state != null) { Map new_state=(Map)Util.objectFromByteBuffer(state); synchronized(service_state) { service_state.clear(); service_state.putAll(new_state); } if(log.isTraceEnabled()) log.trace("service state was set successfully (" + service_state.size() + " entries)"); } else { if(log.isWarnEnabled()) log.warn("received service state was null"); } break; } catch(TimeoutException e) { if(log.isTraceEnabled()) log.trace("timed out waiting for service state from " + coord + ", retrying"); } } } state_req.toString():[dst: 192.168.1.77:7820, src: <null> (1 headers), size = 0 bytes] channel.toString():org.jgroups.JChannel@1977b9b protocolSpecFRAG2 frag_size=65000 FC min_threshold=0.10 max_credits=2000000 GMS shun=true print_local_addr=true view_bundling=true join_timeout=300 join_retry_timeout=2000 STABLE max_bytes=400000 stability_delay=1000 desired_avg_gossip=50000 NAKACK max_xmit_size=60000 retransmit_timeout=300,600,1200,2400,4800 use_mcast_xmit=false discard_delivered_msgs=true gc_lag=0 VERIFY_SUSPECT timeout=1500 FD max_tries=5 shun=true timeout=10000 FD_SOCK MERGE2 max_interval=100000 min_interval=2000 TCPPING port_range=5 num_initial_members=3 initial_hosts=192.168.1.77[7820] timeout=30 TCP discard_incompatible_packets=true sock_conn_timeout=300 bind_addr=192.168.1.77 use_send_queues=false start_port=7820 recv_buf_size=20000000 skip_suspected_members=true send_buf_size=640000 use_incoming_packet_handler=true loopback=fasle chan.toString(treu) True.....local_addr=192.168.1.77:7821 cluster_name=dispatcher_7840 my_view=[192.168.1.77:7820|6] [192.168.1.77:7820, 192.168.1.77:7822, 192.168.1.77:7821] connected=true closed=false incoming queue size=0 receive_blocks=true receive_local_msgs=false auto_reconnect=false auto_getstate=false state_transfer_supported=false props=scheduledtaskexecuteframework.TCP(use_send_queues=false;bind_addr=first-non-local;sock_conn_timeout=300;loopback=fasle;skip_suspected_members=true;discard_incompatible_packets=true;recv_buf_size=20000000;start_port=7820;use_incoming_packet_handler=true;send_buf_size=640000):TCPPING(num_initial_members=3;initial_hosts=192.168.1.77[7820];port_range=5;timeout=30):MERGE2(min_interval=2000;max_interval=100000):FD_SOCK:FD(max_tries=5;timeout=10000;shun=true):VERIFY_SUSPECT(timeout=1500):pbcast.NAKACK(gc_lag=0;use_mcast_xmit=false;retransmit_timeout=300,600,1200,2400,4800;discard_delivered_msgs=true;max_xmit_size=60000):pbcast.STABLE(desired_avg_gossip=50000;max_bytes=400000;stability_delay=1000):pbcast.GMS(print_local_addr=true;view_bundling=true;join_timeout=300;join_retry_timeout=2000;shun=true):FC(max_credits=2000000;min_threshold=0.10):FRAG2(frag_size=65000) chan.toStringorg.jgroups.JChannel@1977b9b service_state_promise is of typeorg.jgroups.util.Promise service_state_promise toString()hasResult=false,result=null _gteResultsWithTimeout8000 _gteResultsWithTimeout start:1181496411298 while timeout, time_to_wait, start, current_time, elapsed8000,8000,1181496411298,1181496411298,0 timeout <=0 doWait(timeout) called doWait(time_to_wait) : 8000 time_to_wait=timeout - (System.currentTimeMillis() - start); -6=8000 - (1181496419304 - 1181496411298) while timeout, time_to_wait, start, current_time, elapsed8000,-6,1181496411298,1181496419304,8006 timeout <=0 doWait(timeout) called ******HURRAY timeout_occurred... -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JGRP-527?page=comments#action_12365289 ] Bryce Alcock commented on JGRP-527: ----------------------------------- I was unable to reproduce it with the DrawMultiplexer as well. However, I was able to consistently reproduce it in my application. I tried 100 tests, and 98 times this event happend. It worked correctly 2 times. Unfortunately I am not really able to publish the proprietary code that I am working with, and in fact I am not sure if I am able to send it in confidence either (I will check on that) A couple of things. 1. I added another use case, where I switch the MuxChannel with a JChannel, for the one specific Mulipexer_ID that was Hanging, and it Never Hung as a JChannel (I have done at least 50 tests). Infact it is working like a champ. Therefore, at this point I am moving forward without the Muliplexer for that channel. 2. I have shown the code block 'public void fetchServiceInformation() throws Exception ...' (posted above) to several engineers and all have agreed that if line 196 throws a timeoutexception, you can end up in an infinite loop which is what it does in my case. So while it might not be reproducible in the case of the simple DrawMultiplexer, it is reproducible in a my application. 3. I am willing to show you my ProtocolStack, and I am willing to add logging statements anywhere you thing useful. But Mainly, I am just looking for Ideas at this point. Also, trying to see if anyone else has had problems like this with the Multiplexer. thanks for the help Let me know how I can proceed -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JGRP-527?page=comments#action_12365628 ] Vladimir Blagojevic commented on JGRP-527: ------------------------------------------ Bryce, Thanks a lot for the test code. Good news: I was able to reproduce issue locally. However, it was very hard for me to follow the application logic of all these multiplexers. I then proceeded to modify all your handlers to make sure that they are not doing anything in their respective receive methods. Thus I isolated application logic from JGroups. After I have done this change I was not able to reproduce this issue any more. This leads me to believe that there is something in application logic that locks up the channel thread. Would you please give it another look and so will I. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ http://jira.jboss.com/jira/browse/JGRP-527?page=comments#action_12365979 ] Bryce Alcock commented on JGRP-527: ----------------------------------- Verified. Good Work, and Nice Catch. Thanks for your help. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira