[infinispan-dev] mux usage

[Ales Justin]

I've changed from 

        dispatcher.sendMessage(message, RequestOptions.ASYNC());

to

        dispatcher.sendMessage(message, RequestOptions.SYNC());

got this now, which might explain why nothing was received before?

23:33:39,725 WARN  [com.arjuna.ats.jta] (http-/192.168.1.101:8080-2) ARJUNA016029: SynchronizationImple.afterCompletion - failed for org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization at 15a0b2f with exception: org.hibernate.search.SearchException: HSEARCH000123: Unable to send Lucene update work via JGroups cluster
	at org.hibernate.search.backend.impl.jgroups.JGroupsBackendQueueTask.sendLuceneWorkList(JGroupsBackendQueueTask.java:97)
	at org.hibernate.search.backend.impl.jgroups.JGroupsBackendQueueProcessor.applyWork(JGroupsBackendQueueProcessor.java:122)
	at org.hibernate.search.indexes.impl.DirectoryBasedIndexManager.performOperations(DirectoryBasedIndexManager.java:127)
	at org.hibernate.search.backend.impl.WorkQueuePerIndexSplitter.commitOperations(WorkQueuePerIndexSplitter.java:61)
	at org.hibernate.search.backend.impl.BatchedQueueingProcessor.performWorks(BatchedQueueingProcessor.java:96)
	at org.hibernate.search.backend.impl.PostTransactionWorkQueueSynchronization.afterCompletion(PostTransactionWorkQueueSynchronization.java:99)
	at com.arjuna.ats.internal.jta.resources.arjunacore.SynchronizationImple.afterCompletion(SynchronizationImple.java:96)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.afterCompletion(TwoPhaseCoordinator.java:402)
	at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.end(TwoPhaseCoordinator.java:103)
	at com.arjuna.ats.arjuna.AtomicAction.commit(AtomicAction.java:164)
	at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1165)
	at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:117)
	at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:75)
	at org.jboss.capedwarf.datastore.JBossTransaction.commit(JBossTransaction.java:149) [capedwarf-datastore-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.jboss.capedwarf.datastore.JBossDatastoreService.put(JBossDatastoreService.java:106) [capedwarf-datastore-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.jboss.capedwarf.datastore.JBossDatastoreService.put(JBossDatastoreService.java:80) [capedwarf-datastore-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.jboss.capedwarf.datastore.JBossDatastoreService.put(JBossDatastoreService.java:76) [capedwarf-datastore-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.jboss.capedwarf.log.JBossLogService.requestStarted(JBossLogService.java:212) [capedwarf-log-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.jboss.capedwarf.appidentity.GAEListener.requestInitialized(GAEListener.java:91) [capedwarf-appidentity-1.0.0-SNAPSHOT.jar:1.0.0-SNAPSHOT]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) [jbossweb-7.0.16.Final.jar:]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.16.Final.jar:]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.16.Final.jar:]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.16.Final.jar:]
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.16.Final.jar:]
	at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:679) [jbossweb-7.0.16.Final.jar:]
	at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:931) [jbossweb-7.0.16.Final.jar:]
	at java.lang.Thread.run(Thread.java:680) [classes.jar:1.6.0_31]
Caused by: org.jgroups.TimeoutException: timeout sending message to node-b/capedwarf
	at org.jgroups.blocks.MessageDispatcher.sendMessage(MessageDispatcher.java:360)
	at org.hibernate.search.backend.impl.jgroups.DispatcherMessageSender.send(DispatcherMessageSender.java:45)
	at org.hibernate.search.backend.impl.jgroups.JGroupsBackendQueueTask.sendLuceneWorkList(JGroupsBackendQueueTask.java:91)
	... 26 more

How to track down timeout cause?

Although from diff stack-trace, I see this:

	at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:202)
	at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:456) [jgroups-3.0.9.Final.jar:3.0.9.Final]
	at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:363) [jgroups-3.0.9.Final.jar:3.0.9.Final]
	at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:238) [jgroups-3.0.9.Final.jar:3.0.9.Final]
	at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:543) [jgroups-3.0.9.Final.jar:3.0.9.Final]
	at org.jgroups.blocks.mux.MuxUpHandler.up(MuxUpHandler.java:130) [jgroups-3.0.9.Final.jar:3.0.9.Final]
	at org.jgroups.JChannel.up(JChannel.java:716) [jgroups-3.0.9.Final.jar:3.0.9.Final]
	at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1026) [jgroups-3.0.9.Final.jar:3.0.9.Final]
	at org.jgroups.protocols.RSVP.up(RSVP.java:179) [jgroups-3.0.9.Final.jar:3.0.9.Final]

        public Object up(Event evt) {
            if(corr != null) {
                if(!corr.receive(evt)) {
                    try {
                        return handleUpEvent(evt);

Meaning msg listener only get's hit if corr::receive returns false.
Is this the case ever - with Ispans handles in place - here?

-Ales

On May 14, 2012, at 11:26 PM, Ales Justin wrote:

> Could be that I don't fully understand how this mux stuff should work. :-)
> As we've been busting heads since Friday ... 
> 
> This is the code (see below) that sets up dispatcher, and sets the listener.
> 
> As you can see I add the msg listener to dispatcher.
> And I use dispatcher's ::getProtocolAdapter to setup UpHandler for mux_id.
> 
> I then expect the following:
> * slave tries sending the msg to master
> * this dispatcher adds mux_id / scopeId to the msg
> * on master, dispatcher's msg listener should pick up this msg
> 
> The problem is that our msg listener is never hit,
> meaning no slave msgs are ever received.
> 
> Any idea what we're doing wrong or what's missing?
> 
> -Ales
> 
> ---
> 
>        UpHandler handler = channel.getUpHandler();
>        if (handler instanceof Muxer) {
>            Short n = (Short) props.get(MUX_ID);
>            if (n == null) {
>                throw new IllegalArgumentException("Missing mux id!");
>            }
>            @SuppressWarnings("unchecked")
>            Muxer<UpHandler> muxer = (Muxer<UpHandler>) handler;
>            if (muxer.get(n) != null) {
>                throw new IllegalArgumentException("Muxer with id " + n + " already used!");
>            }
> 
>            muxId = n;
>            ClassLoader cl = (ClassLoader) props.get(CLASSLOADER);
>            MessageListener wrapper = (cl != null) ? new ClassloaderMessageListener(listener, cl) : listener;
>            MessageDispatcher dispatcher = new MuxMessageDispatcher(muxId, channel, wrapper, listener, null);
>            muxer.add(muxId, dispatcher.getProtocolAdapter());
>            sender = new DispatcherMessageSender(dispatcher);
>