4:26 p.m.
Could be that I don't fully understand how this mux stuff should work. :-)
As we've been busting heads since Friday ...
This is the code (see below) that sets up dispatcher, and sets the listener.
As you can see I add the msg listener to dispatcher.
And I use dispatcher's ::getProtocolAdapter to setup UpHandler for mux_id.
I then expect the following:
* slave tries sending the msg to master
* this dispatcher adds mux_id / scopeId to the msg
* on master, dispatcher's msg listener should pick up this msg
The problem is that our msg listener is never hit,
meaning no slave msgs are ever received.
Any idea what we're doing wrong or what's missing?
-Ales
---
UpHandler handler = channel.getUpHandler();
if (handler instanceof Muxer) {
Short n = (Short) props.get(MUX_ID);
if (n == null) {
throw new IllegalArgumentException("Missing mux id!");
}
@SuppressWarnings("unchecked")
Muxer<UpHandler> muxer = (Muxer<UpHandler>) handler;
if (muxer.get(n) != null) {
throw new IllegalArgumentException("Muxer with id " + n + "
already used!");
}
muxId = n;
ClassLoader cl = (ClassLoader) props.get(CLASSLOADER);
MessageListener wrapper = (cl != null) ? new
ClassloaderMessageListener(listener, cl) : listener;
MessageDispatcher dispatcher = new MuxMessageDispatcher(muxId, channel,
wrapper, listener, null);
muxer.add(muxId, dispatcher.getProtocolAdapter());
sender = new DispatcherMessageSender(dispatcher);
4:51 p.m.
I've changed from
dispatcher.sendMessage(message, RequestOptions.ASYNC());
to
dispatcher.sendMessage(message, RequestOptions.SYNC());
got this now, which might explain why nothing was received before?
23:33:39,725 WARN [com.arjuna.ats.jta] (http-/192.168.1.101:8080-2) ARJUNA016029:
SynchronizationImple.afterCompletion - failed for
org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization@15a0b2f with
exception: org.hibernate.search.SearchException: HSEARCH000123: Unable to send Lucene
update work via JGroups cluster
at
org.hibernate.search.backend.impl.jgroups.JGroupsBackendQueueTask.sendLuceneWorkList(JGroupsBackendQueueTask.java:97)
at
org.hibernate.search.backend.impl.jgroups.JGroupsBackendQueueProcessor.applyWork(JGroupsBackendQueueProcessor.java:122)
at
org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.performOperations(DirectoryBasedIndexManager.java:127)
at
org.hibernate.search.backend.impl.WorkQueuePerIndexSplitter.commitOperations(WorkQueuePerIndexSplitter.java:61)
at
org.hibernate.search.backend.impl.BatchedQueueingProcessor.performWorks(BatchedQueueingProcessor.java:96)
at
org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization.afterCompletion(PostTransactionWorkQueueSynchronization.java:99)
at
com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
at
com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103)
at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)
at
com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
at
com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
at
com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
at org.jboss.capedwarf.datastore.JBossTransaction.commit(JBossTransaction.java:149)
[capedwarf-datastore-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at
org.jboss.capedwarf.datastore.JBossDatastoreService.put(JBossDatastoreService.java:106)
[capedwarf-datastore-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at org.jboss.capedwarf.datastore.JBossDatastoreService.put(JBossDatastoreService.java:80)
[capedwarf-datastore-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at org.jboss.capedwarf.datastore.JBossDatastoreService.put(JBossDatastoreService.java:76)
[capedwarf-datastore-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at org.jboss.capedwarf.log.JBossLogService.requestStarted(JBossLogService.java:212)
[capedwarf-log-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at org.jboss.capedwarf.appidentity.GAEListener.requestInitialized(GAEListener.java:91)
[capedwarf-appidentity-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
[jbossweb-7.0.16.Final.jar:]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
[jbossweb-7.0.16.Final.jar:]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
[jbossweb-7.0.16.Final.jar:]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368)
[jbossweb-7.0.16.Final.jar:]
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877)
[jbossweb-7.0.16.Final.jar:]
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:679)
[jbossweb-7.0.16.Final.jar:]
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931)
[jbossweb-7.0.16.Final.jar:]
at java.lang.Thread.run(Thread.java:680) [classes.jar:1.6.0_31]
Caused by: org.jgroups.TimeoutException: timeout sending message to node-b/capedwarf
at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:360)
at
org.hibernate.search.backend.impl.jgroups.DispatcherMessageSender.send(DispatcherMessageSender.java:45)
at
org.hibernate.search.backend.impl.jgroups.JGroupsBackendQueueTask.sendLuceneWorkList(JGroupsBackendQueueTask.java:91)
... 26 more
How to track down timeout cause?
Although from diff stack-trace, I see this:
at
org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:202)
at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456)
[jgroups-3.0.9.Final.jar:3.0.9.Final]
at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363)
[jgroups-3.0.9.Final.jar:3.0.9.Final]
at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238)
[jgroups-3.0.9.Final.jar:3.0.9.Final]
at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543)
[jgroups-3.0.9.Final.jar:3.0.9.Final]
at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130)
[jgroups-3.0.9.Final.jar:3.0.9.Final]
at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.9.Final.jar:3.0.9.Final]
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026)
[jgroups-3.0.9.Final.jar:3.0.9.Final]
at org.jgroups.protocols.RSVP.up(RSVP.java:179) [jgroups-3.0.9.Final.jar:3.0.9.Final]
public Object up(Event evt) {
if(corr != null) {
if(!corr.receive(evt)) {
try {
return handleUpEvent(evt);
Meaning msg listener only get's hit if corr::receive returns false.
Is this the case ever - with Ispans handles in place - here?
-Ales
On May 14, 2012, at 11:26 PM, Ales Justin wrote:
Could be that I don't fully understand how this mux stuff should
work. :-)
As we've been busting heads since Friday ...
This is the code (see below) that sets up dispatcher, and sets the listener.
As you can see I add the msg listener to dispatcher.
And I use dispatcher's ::getProtocolAdapter to setup UpHandler for mux_id.
I then expect the following:
* slave tries sending the msg to master
* this dispatcher adds mux_id / scopeId to the msg
* on master, dispatcher's msg listener should pick up this msg
The problem is that our msg listener is never hit,
meaning no slave msgs are ever received.
Any idea what we're doing wrong or what's missing?
-Ales
---
UpHandler handler = channel.getUpHandler();
if (handler instanceof Muxer) {
Short n = (Short) props.get(MUX_ID);
if (n == null) {
throw new IllegalArgumentException("Missing mux id!");
}
@SuppressWarnings("unchecked")
Muxer<UpHandler> muxer = (Muxer<UpHandler>) handler;
if (muxer.get(n) != null) {
throw new IllegalArgumentException("Muxer with id " + n + "
already used!");
}
muxId = n;
ClassLoader cl = (ClassLoader) props.get(CLASSLOADER);
MessageListener wrapper = (cl != null) ? new
ClassloaderMessageListener(listener, cl) : listener;
MessageDispatcher dispatcher = new MuxMessageDispatcher(muxId, channel,
wrapper, listener, null);
muxer.add(muxId, dispatcher.getProtocolAdapter());
sender = new DispatcherMessageSender(dispatcher);
4:13 p.m.
public Object up(Event evt) {
if(corr != null) {
if(!corr.receive(evt)) {
try {
return handleUpEvent(evt);
Meaning msg listener only get's hit if corr::receive returns false.
Is this the case ever - with Ispans handles in place - here?
I think it managed to see why no listener kicks in -- if my debug was right.
And the above code -- as suspected -- is the culprit.
corr.receive invokes
case Event.MSG:
if(receiveMessage((Message)evt.getArg()))
return true; // message was consumed, don't pass it up
Where "receiveMessage" goes here:
switch(hdr.type) {
case Header.REQ:
if(request_handler == null) {
if(log.isWarnEnabled()) {
log.warn("there is no request handler installed to deliver
request !");
}
return true;
}
handleRequest(msg, hdr); <----------------------- HERE
break;
..... // other code
return true; // message was consumed
----
Our Msg dispatcher is the request_handler,
but since we don't setup any explicit hander on it, it does nothing, retval == null.
And then
if(!hdr.rsp_expected) // asynchronous call, we don't need to send a response;
terminate call here
return;
is true, since I put back async usage, so we return nicely.
If you go back "receiveMessage" on ReqCorr returns true, hence we think msg was
properly consumed,
so no need for listener to handle it.
Bug in JGroups?
---
protected void handleRequest(Message req, Header hdr) {
Object retval;
Object rsp_buf; // either byte[] or Buffer
Header rsp_hdr;
Message rsp;
boolean threw_exception=false;
// i. Get the request correlator header from the msg and pass it to
// the registered handler
//
// ii. If a reply is expected, pack the return value from the request
// handler to a reply msg and send it back. The reply msg has the same
// ID as the request and the name of the sender request correlator
if(log.isTraceEnabled()) {
log.trace(new StringBuilder("calling (").append((request_handler !=
null? request_handler.getClass().getName() : "null")).
append(") with request ").append(hdr.id));
}
try {
retval=request_handler.handle(req);
}
catch(Throwable t) {
// if(log.isErrorEnabled()) log.error("error invoking method", t);
threw_exception=true;
retval=t;
}
if(!hdr.rsp_expected) // asynchronous call, we don't need to send a response;
terminate call here
return;
4:25 p.m.
Ha, I was right. ;-)
Adding this code:
MessageListenerToRequestHandlerAdapter adapter = new
MessageListenerToRequestHandlerAdapter(wrapper);
MessageDispatcher dispatcher = new MuxMessageDispatcher(muxId, channel,
wrapper, listener, adapter);
does the job -- listener gets invoked now.
So, the question is, JGroups bug, or intentional?
-Ales
On May 15, 2012, at 11:13 PM, Ales Justin wrote:
>
> public Object up(Event evt) {
> if(corr != null) {
> if(!corr.receive(evt)) {
> try {
> return handleUpEvent(evt);
>
> Meaning msg listener only get's hit if corr::receive returns false.
> Is this the case ever - with Ispans handles in place - here?
I think it managed to see why no listener kicks in -- if my debug was right.
And the above code -- as suspected -- is the culprit.
corr.receive invokes
case Event.MSG:
if(receiveMessage((Message)evt.getArg()))
return true; // message was consumed, don't pass it up
Where "receiveMessage" goes here:
switch(hdr.type) {
case Header.REQ:
if(request_handler == null) {
if(log.isWarnEnabled()) {
log.warn("there is no request handler installed to deliver
request !");
}
return true;
}
handleRequest(msg, hdr); <----------------------- HERE
break;
..... // other code
return true; // message was consumed
----
Our Msg dispatcher is the request_handler,
but since we don't setup any explicit hander on it, it does nothing, retval == null.
And then
if(!hdr.rsp_expected) // asynchronous call, we don't need to send a response;
terminate call here
return;
is true, since I put back async usage, so we return nicely.
If you go back "receiveMessage" on ReqCorr returns true, hence we think msg was
properly consumed,
so no need for listener to handle it.
Bug in JGroups?
---
protected void handleRequest(Message req, Header hdr) {
Object retval;
Object rsp_buf; // either byte[] or Buffer
Header rsp_hdr;
Message rsp;
boolean threw_exception=false;
// i. Get the request correlator header from the msg and pass it to
// the registered handler
//
// ii. If a reply is expected, pack the return value from the request
// handler to a reply msg and send it back. The reply msg has the same
// ID as the request and the name of the sender request correlator
if(log.isTraceEnabled()) {
log.trace(new StringBuilder("calling (").append((request_handler !=
null? request_handler.getClass().getName() : "null")).
append(") with request ").append(hdr.id));
}
try {
retval=request_handler.handle(req);
}
catch(Throwable t) {
// if(log.isErrorEnabled()) log.error("error invoking method", t);
threw_exception=true;
retval=t;
}
if(!hdr.rsp_expected) // asynchronous call, we don't need to send a response;
terminate call here
return;
6:29 a.m.
Hi Ales,
I looked into this, and - yes - a MessageListener passed to an
RpcDispatcher or MessageDispatcher or MuxRpcDispatcher (subclass of
RpcDispatcher) is ignored.
This is intentional; or else a MessageListener's receive() callback
could eat the message destined for the internal Receiver, which calls
handleMessage() in case of MessageDispatcher or invokes a method in case
of the RpcDispatcher.
I'll create a JIRA to remove the MessageListener from
{Rpc/Message}Dispatcher.
What are you trying to do ? Why don't you simply implement a method you
want to call (say foo()) and then invoke foo() from your caller ?
On 5/15/12 11:13 PM, Ales Justin wrote:
>
> public Object up(Event evt) {
> if(corr != null) {
> if(!corr.receive(evt)) {
> try {
> return handleUpEvent(evt);
>
> Meaning msg listener only get's hit if corr::receive returns false.
> Is this the case ever - with Ispans handles in place - here?
I think it managed to see why no listener kicks in -- if my debug was right.
And the above code -- as suspected -- is the culprit.
corr.receive invokes
case Event.MSG:
if(receiveMessage((Message)evt.getArg()))
return true; // message was consumed, don't pass it up
Where "receiveMessage" goes here:
switch(hdr.type) {
case Header.REQ:
if(request_handler == null) {
if(log.isWarnEnabled()) {
log.warn("there is no request handler installed to deliver
request !");
}
return true;
}
handleRequest(msg, hdr);<----------------------- HERE
break;
..... // other code
return true; // message was consumed
----
Our Msg dispatcher is the request_handler,
but since we don't setup any explicit hander on it, it does nothing, retval == null.
And then
if(!hdr.rsp_expected) // asynchronous call, we don't need to send a
response; terminate call here
return;
is true, since I put back async usage, so we return nicely.
If you go back "receiveMessage" on ReqCorr returns true, hence we think msg was
properly consumed,
so no need for listener to handle it.
Bug in JGroups?
---
protected void handleRequest(Message req, Header hdr) {
Object retval;
Object rsp_buf; // either byte[] or Buffer
Header rsp_hdr;
Message rsp;
boolean threw_exception=false;
// i. Get the request correlator header from the msg and pass it to
// the registered handler
//
// ii. If a reply is expected, pack the return value from the request
// handler to a reply msg and send it back. The reply msg has the same
// ID as the request and the name of the sender request correlator
if(log.isTraceEnabled()) {
log.trace(new StringBuilder("calling (").append((request_handler
!= null? request_handler.getClass().getName() : "null")).
append(") with request ").append(hdr.id));
}
try {
retval=request_handler.handle(req);
}
catch(Throwable t) {
// if(log.isErrorEnabled()) log.error("error invoking method",
t);
threw_exception=true;
retval=t;
}
if(!hdr.rsp_expected) // asynchronous call, we don't need to send a
response; terminate call here
return;
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Bela Ban, JGroups lead (http://www.jgroups.org)
6:33 a.m.
Correction to my previous email: MessageListener *is* used, but only its
getState() and setState() methods will ever be called. Its receive()
method will *never* be called.
On 5/15/12 11:13 PM, Ales Justin wrote:
>
> public Object up(Event evt) {
> if(corr != null) {
> if(!corr.receive(evt)) {
> try {
> return handleUpEvent(evt);
>
> Meaning msg listener only get's hit if corr::receive returns false.
> Is this the case ever - with Ispans handles in place - here?
I think it managed to see why no listener kicks in -- if my debug was right.
And the above code -- as suspected -- is the culprit.
corr.receive invokes
case Event.MSG:
if(receiveMessage((Message)evt.getArg()))
return true; // message was consumed, don't pass it up
Where "receiveMessage" goes here:
switch(hdr.type) {
case Header.REQ:
if(request_handler == null) {
if(log.isWarnEnabled()) {
log.warn("there is no request handler installed to deliver
request !");
}
return true;
}
handleRequest(msg, hdr);<----------------------- HERE
break;
..... // other code
return true; // message was consumed
----
Our Msg dispatcher is the request_handler,
but since we don't setup any explicit hander on it, it does nothing, retval == null.
And then
if(!hdr.rsp_expected) // asynchronous call, we don't need to send a
response; terminate call here
return;
is true, since I put back async usage, so we return nicely.
If you go back "receiveMessage" on ReqCorr returns true, hence we think msg was
properly consumed,
so no need for listener to handle it.
Bug in JGroups?
---
protected void handleRequest(Message req, Header hdr) {
Object retval;
Object rsp_buf; // either byte[] or Buffer
Header rsp_hdr;
Message rsp;
boolean threw_exception=false;
// i. Get the request correlator header from the msg and pass it to
// the registered handler
//
// ii. If a reply is expected, pack the return value from the request
// handler to a reply msg and send it back. The reply msg has the same
// ID as the request and the name of the sender request correlator
if(log.isTraceEnabled()) {
log.trace(new StringBuilder("calling (").append((request_handler
!= null? request_handler.getClass().getName() : "null")).
append(") with request ").append(hdr.id));
}
try {
retval=request_handler.handle(req);
}
catch(Throwable t) {
// if(log.isErrorEnabled()) log.error("error invoking method",
t);
threw_exception=true;
retval=t;
}
if(!hdr.rsp_expected) // asynchronous call, we don't need to send a
response; terminate call here
return;
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--
Bela Ban, JGroups lead (http://www.jgroups.org)
5069
days inactive
5071
days old
infinispan-dev@lists.jboss.org
5 comments
2 participants
participants (2)
-
Ales Justin -
Bela Ban