[jboss-jira] [JBoss JIRA] Commented: (JGRP-1083) TP: remove sending from critical path

TP: remove sending from critical path ------------------------------------- Key: JGRP-1083 URL: https://jira.jboss.org/jira/browse/JGRP-1083 Project: JGroups Issue Type: Task Reporter: Bela Ban Assignee: Bela Ban Fix For: 2.10 Attachments: TP.java.good In TP.send(), we (1) marshall a message (or message list) and (2) send it. Both (1) and (2) are on the critical path, regardless of whether we send the message directly, or via the Bundler. In the latter case, when we've filled the queue, we actually marshall a *list* of messages, which adds to the time JChannel.send() takes. (2) is the lock we acquire before marshalling and sending a message. When we receive a message, oftentimes, we have to send a message in the same thread which handles the incoming message. This means, we incur (1) and (2) before we return and process the next message, therefore delaying message processing (at least for non-OOB messages from the same sender). E.g. in UNICAST.up(), we send an ACK before we return and handle the next message. Sending the ACK entails marshalling the message and sending it, which add to the processing time of a unicast message. Similar cases are NAKACK retransmissions and responses to requests. To remove this time from the critical path, I suggest we do the following things: #1 Remove TP.lock and provide output streams for each message The cost of creating an output stream (ExposedByteArrayOutputStream and ExposedDataOutputStream) is small, so we can do it for every message. In the bundler, we create it once and then use it for marshalling of the different lists in the hashmap (with a reset() before each list). In addition, this won't keep allocated buffers around. E.g. if we send a large message in TCP (say 2MB), the BAOS's buffer will remain at 2MB and not go back even if we send smaller messages afterwards ! #2 Send the messages in a separate thread To eliminate (1) and (2) from the critical path, we simply send certain messages in a separate thread. Marshalling and sending is now done on a separate thread, so e.g. UNICAST.up() will send the ACK in a separate thread and return immediately, removing the time to marshall and send the ACK from the processing time of the unicast message. We could either introduce a sender thread pool (with a bounded queue) and rejection policy "run", or reuse the TimeScheduler. The issue with the latter is though that retransmissions already in the TimeScheduler would create a new task in the TimeScheduler for every retransmission... We would need to tag messages to be sent asynchronously, or reuse the OOB tag (TBD). Or we could add a downAsync(Event evt) method to Protocol... #3 Optimize Bundler We have a coarse grained lock here, which locks both access to variables (like count) and the message hashmap. Investigate 2 separate locks, one for state vars and one for accessing the hashmap.