Initially some close() methods were missing but I added them and still it was the issue. (BiSockets getting collected). Then I realised for every message, InitialContext and ConnectionFactory are being instantiated. But all close() methods were called.
I changed the code to get InitialContext and ConnectionFactory once per queue. Now the BisocketServerInvokers are limited in number. But I still did get why BisocketServerInvokers are getting accumulated when I open 1 InitialContext and ConnectoryFactory per message, but still calling all close().