[JBoss JIRA] Created: (JBMESSAGING-1456) Messages stuck in being-delivered state in cluster
by Justin Bertram (JIRA)
Messages stuck in being-delivered state in cluster
--------------------------------------------------
Key: JBMESSAGING-1456
URL: https://jira.jboss.org/jira/browse/JBMESSAGING-1456
Project: JBoss Messaging
Issue Type: Bug
Reporter: Justin Bertram
Assignee: Tim Fox
Priority: Critical
Messages become "stuck" in being-delivered state when clients use a clustered XA connection factory in a cluster of at least 2 nodes.
JBoss setup:
-2 nodes of JBoss EAP 4.3 CP02
-commented out "ClusterPullConnectionFactory" in messaging-service.xml to prevent message redistribution and eliminate the "message suckers" as the potential culprit
-MySQL backend using the default mysql-persistence-service.xml (from <JBOSS_HOME>/docs/examples/jms)
Client setup:
-both nodes have a client which is a separate process (i.e. not inside JBoss)
-clients are Spring based
-one client produces and consumes, the other client just consumes
-both clients use the ClusteredXAConnectionFactory from the default connection-factories-service.xml
-both clients publish to and consume from "queue/testDistributedQueue"
-clients are configured to send persistent messages, use AUTO_ACKNOWLEDGE, and transacted sessions
Symptoms of the issue:
-when running the clients I watch the JMX-Console for the "queue/testDistributedQueue"
-as the consumers pull messages off the queue I can see the MessageCount and DeliveringCount go to 0 every so often
-after a period of time (usually a few hours) the MessageCount and DeliveringCount never go back to 0
-I "kill" the clients and wait for the DeliveringCount to go to 0, but it never does
-after the clients are killed the ConsumerCount for the queue will drop, but never to 0 when messages are "stuck"
-a thread dump reveals at least one JBM server session that is apparently stuck (it never goes away) - ostensibly this is the consumer that is showing in the JMX-Console for "queue/testDistributedQueue"
-a "killall -3 java" doesn't produce anything from the clients so I know their dead
-nothing is in any DLQ or expiry queue
-the database contains as many rows in the JBM_MSG and JBM_MSG_REF tables as the DeliveringCount in the JMX-Console
-rebooting the node with the stuck messages frees the messages to be consumed (i.e. un-sticks them)
Other notes:
-nothing else is happening on either node but running the client and running JBoss
-this only appears to happen when a clustered connection factory is used. I tested using a normal connection factory and after 24 hours couldn't reproduce a stuck message.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
14 years, 3 months
[JBoss JIRA] Created: (EJBTHREE-1330) EJB timer service should use a thread pool to avoid OOM
by Galder Zamarreno (JIRA)
EJB timer service should use a thread pool to avoid OOM
-------------------------------------------------------
Key: EJBTHREE-1330
URL: http://jira.jboss.com/jira/browse/EJBTHREE-1330
Project: EJB 3.0
Issue Type: Bug
Components: pool
Affects Versions: AS 4.2.2.GA
Reporter: Galder Zamarreno
Assigned To: Galder Zamarreno
Priority: Minor
The default EJB timer service used by the EJB3 layer is based on
org.jboss.ejb3.timerservice.jboss.JBossTimerServiceFactory which delegates
to the standard org.jboss.ejb.txtimer.EJBTimerService.
For EJB3 beans using the EJB timer service the thread local pool should not be used.
Since the current EJB timer service creates a new thread for each timer being created, the
thread local pool will create a matching instance of the bean for that thread. Thus the number
of active instances in total can effectively grow unchecked and thus an OOM will occur.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
14 years, 3 months
[JBoss JIRA] Created: (JBREM-552) cannot init cause of ClassCastException
by John Mazzitelli (JIRA)
cannot init cause of ClassCastException
---------------------------------------
Key: JBREM-552
URL: http://jira.jboss.com/jira/browse/JBREM-552
Project: JBoss Remoting
Issue Type: Bug
Security Level: Public (Everyone can see)
Affects Versions: 2.0.0.Beta2 (Boon)
Reporter: John Mazzitelli
Assigned To: Tom Elrod
Priority: Minor
Fix For: 2.0.0.CR1 (Boon)
I'm in a catch clause within InvokerRegistry and I'm getting a weird exception thrown:
java.lang.IllegalStateException: Can't overwrite cause
at java.lang.Throwable.initCause(Throwable.java:320)
at org.jboss.remoting.InvokerRegistry.loadClientInvoker(InvokerRegistry.java:447)
at org.jboss.remoting.InvokerRegistry.createClientInvoker(InvokerRegistry.java:324)
at org.jboss.remoting.Client.connect(Client.java:385)
at org.jboss.on.communications.command.client.JBossRemotingRemoteCommunicator.getRemotingClient(JBossRemotingRemoteCommunicator.java:470)
at org.jboss.on.communications.command.client.JBossRemotingRemoteCommunicator.send(JBossRemotingRemoteCommunicator.java:430)
at org.jboss.on.communications.command.client.AbstractCommandClient.invoke(AbstractCommandClient.java:167)
at org.jboss.on.communications.command.client.ClientCommandSender.send(ClientCommandSender.java:820)
at org.jboss.on.communications.command.client.ServerPollingThread.run(ServerPollingThread.java:102)
Look at line 447 and you'll see it is trying to init the cause of a ClassNotFoundException. Running in a debugger, the newly constructed exception created on 446 has a null cause. Looking then at Throwable.initCause and you'll see a null cause causes this IllegalStateException to be thrown. The cause needs to be "this", not null (I don't know why, seems like it should also look for null, but whatever - that's the way the JDK is written).
The fix is simple - use the constructor that takes a throwable as its second parameter. Not sure why initCause it being used, as opposed to this constructor.
e.g.:
new ClassNotFoundException("Can not invoke loadClientInvokerClass method on " + transportFactoryClass, e);
(notice the ", e" parameter).
There are three other instances where initCause is called on ClassNotFoundException that also need to be fixed. See the other catch clauses in here.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://jira.jboss.com/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
14 years, 5 months
[JBoss JIRA] Created: (JGRP-809) Copyless stack
by Bela Ban (JIRA)
Copyless stack
--------------
Key: JGRP-809
URL: https://jira.jboss.org/jira/browse/JGRP-809
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Currently (as of 2.7), the transport reads the contents of a received packet into a buffer, then passes a *copy* of the buffer to a thread from the OOB or incoming thread pools. To prevent this copy, we can
- have the receiver read only the version and OOB flag (to see which thread pool to dispatch the packet to)
- pass a ref to the socket to a thread from the incoming of OOB pool, have that thread read the packet and return
- each thread in the pool has its own buffer into which the buffer is read from the socket
Possibly use NIO: we can install a selector and get woken up whenever data to be read is present. At that point, we can pass the ref to the socket to the handler thread and return immediately. NIO with channels for multicast sockets is available only in JDK 7 (or 8?), so this is a bit off... However, we can already implement this with reading the version and flags bytes and then passing the socket to the handler
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
14 years, 5 months
[JBoss JIRA] Created: (JGRP-815) Scatter/Gather to avoid copying
by Bela Ban (JIRA)
Scatter/Gather to avoid copying
-------------------------------
Key: JGRP-815
URL: https://jira.jboss.org/jira/browse/JGRP-815
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
When we invoke Channel.send(), we pass a bufffer to JGroups. At the transport level, JGroups marshals the sender and destination address, plus all headers and the buffer into a new byte[] buffer, which is then passed to the socket (DatagramSocket, MulticastSocket, Socket).
We cannot do gathering writes on a DatagramSocket because DatagramSocket doesn't expose this functionality, contrary to a DatagramChannel.
We could avoid having to copy the user's buffer by using gathering writes: effectively passing to the socket NIO ByteBuffers containing:
1: Src and dest address plus flags, plus possibly size
2: The marshalled headers
3: The buffer passed to JGroups by the user
We can obtain a gathering-write channel as follows:
ByteBuffer[] buffers; // contains the 3 byte buffers above
DatagramSocket sock;
DatagramChannel ch=sock.getChannel();
ch.write(buffers, 0, length); // length is the number of bytes of the total marshalled message
This is supported by a GatheringByteChannel.
I don't think there's currently a need to do scattered reads, but this needs to get investigated more. Also investigate whether MulticastSockets support gathering writes (whether they expose the correct DatagramChannel).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
14 years, 5 months
[JBoss JIRA] Created: (JGRP-816) TP: avoid copying when receiving data
by Bela Ban (JIRA)
TP: avoid copying when receiving data
-------------------------------------
Key: JGRP-816
URL: https://jira.jboss.org/jira/browse/JGRP-816
Project: JGroups
Issue Type: Feature Request
Reporter: Bela Ban
Assignee: Bela Ban
Fix For: 2.8
Currently, we receive data into a byte[] buffer, e.g. from DatagramSocket.receive(), but then have to COPY buffer when passing it to a worker from the thread pool. The copy is needed because the next DatargamSocket.receive() will overwrite the contents of buffer, and if the worker thread is still unmarshalling the data from the buffer, it might read corrupt contents.
To overcome this, investigate the following:
- Every worker thread has it own buffer
- When the thread has been idle for N seconds and is removed from the pool, that buffer will be discarded
- We create DatagramChannels from the socket(s) (DatagramSocket, MulticastSocket or Socket)
- A selector is called when new data is available on the socket
- The key returned by the selector points to the right channel
- We pass the channel to a thread pool worker thread and continue with the select() loop
- The worker then reads the data into its own buffer, unmarshalls it and passed it up the stack
==> No copy of the buffer is required as the thread's buffer is available until the message has been processed (usually until message unmarshalling)
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
14 years, 5 months