I found this bug (pretty sure about it being a bug, would love to directly file an issue in the issue tracker, but that doesn't seem to be possible without having some SalesForce "support case reference") when porting a real application which explicitly supports offline scenarios from JBoss 6 to 7.1.3:
When an EJB client gets disconnected from its JBoss 7 server, for example because that server has been shut down, but keeps running and eventually reconnects to the restarted server, a new org.jboss.remoting3.ConnectionImpl instance is created for that new connection. However, the old connection (with its state now being CLOSED) can never be garbage collected, since it is still referenced in the set "pendingInboundConnections" within its RemoteConnectionProvider (actually there are two more references to it preventing a GC, but those are in jboss-ejb-client, so they're probably off-topic in this discussion board - I'll start a similar thread there to get these addressed).
The connection is not removed from that set during its entire lifetime, neither after successfully establishing the connection nor after it has been closed. Thus, frequent disconnects between an EJB client and server eventually fill up the heap of the client with closed connections.
I attempted to come up with a solution to remove these dead connections from the set and found one which seems to be working in our application, albeit it's probably not the most elegant one; a patch file is attached to this posting.
Steps to reproduce the issue:
- Create an EJB client and service, deploy service in JBoss7
- Have the EJB client call the service on the server frequently
- Shut down the server. The client must be built to continue calling the server, even though exceptions are now thrown.
- Restart the server. The client will reconnect and continue without errors.
- Fetch a heap dump from the client and load it up in your favorite analyzer tool. There are now two connection objects in the heap, a closed one and an open one. Rinse and repeat for having 3,4,5,...
I can reproduce it most easily with the JBoss ejb-remote quickstart sample, modified to call the server repeatedly. The reference mentioned above is clearly visible in the heap dump.
The bug is definitely present in the most current final of Remoting 3, v3.2.16.GA, which I based my little patch on.