Sending large files

[Jiang Bian]

Thank you for the quick response.

It's my server and I am pretty sure the format of the command is
right. I was trying to see the throughput difference between UDP and
TCP, so I did a UDP version as well, which works pretty well and
stable. So, I think the decode part should be fine (only a few changes
needed from UDP to TCP, such as change ChannelBuffer.clear() to
ChannelBuffer.discardReadBytes(), etc... applaud for netty's universal
interface!!!)

The code works fine with small files, but hangs when sending large
ones. I do see the "end chunk data" in the log, but it just sits there
after that. I'll try the wireshark to see if the command really sends
out, but my guess is it didn't, which I totally don't understand
why...

 The following is the part of the server code.

 @Override
    public void messageReceived(ChannelHandlerContext ctx, MessageEvent e)
	    throws Exception {

	this.bytesReceived.addAndGet(((ChannelBuffer) e.getMessage())
		.readableBytes());

	ChannelBuffer m = (ChannelBuffer) e.getMessage();
	buf.writeBytes(m);

// a JigDFS command is 8 bytes
	if (buf.readableBytes() >= 8) {
	    buf.markReaderIndex();

	    //logger.debug("" + m.readableBytes());
	    int jigDFSCommand = JigDFSCommandDecodeHelper.processInput(buf);
	
	    switch (jigDFSCommand) {
	    case JigDFSCommand.END_CHUNK_DATA: {
		logger.debug("END_CHUNK_DATA COMMAND");
		System.out.println("whatever");
		System.exit(-1);
		FileChunkJoiner fcJoiner = (FileChunkJoiner) getObjectInNode(e.getChannel()
			.getRemoteAddress().toString(), "FileChunkJoiner");
		
		
		if (fcJoiner == null) {
		    logger
			    .warn("i am not waiting for a file... why am i getting a chunk data...");
		
		    logger.debug(e.getChannel().getRemoteAddress().toString());
		    System.exit(-1);
		
		    buf.resetReaderIndex();
		    break;
		}

		if (!fcJoiner.isCompleted()) {
		    // still have segments needed...
		    logger.debug("still have " + fcJoiner.getSegmentNeeded().size()
+ " segments");
		    ChannelBuffer buffer = JigDFSProtocolRequestFactory
			    .reqSegments(fcJoiner.getSegmentNeeded());

		    this.bytesSent.addAndGet(buffer.capacity());
		    e.getChannel().write(buffer);

		} else {
		    logger.debug("file is complete!");
		    if (!fcJoiner.verifyIntegrity()) {
			logger.error("however it's not intact!");
		    }
		
		    SaveSlice saveSlice = (SaveSlice) getObjectInNode(e.getChannel()
				.getRemoteAddress().toString(), "SaveSlice");
		
		    SimulationHelper.getSimulationHelper().insertSimulationData(this.node,
saveSlice.getSliceLength(), saveSlice.getFileID(),
saveSlice.getSliceID(), System.currentTimeMillis() / 1000,
this.bytesReceived.get(), this.bytesSent.get(), "SAVE_SLICE_END");
			
		    removeObjectforNode(e.getChannel().getRemoteAddress().toString(),
"SaveSlice");
		    removeObjectforNode(e.getChannel().getRemoteAddress().toString(),
"FileChunkJoiner");
		    //do i care? if the client doesn't close the
channel?.awaitUninterruptibly()?
		    e.getChannel().write(JigDFSProtocolRequestFactory.endSaveSlice());
		}

		buf.discardReadBytes();

	    }
	
		break;

         ..............
       }

Thanks,

Jiang

And yes, it's tcp. I had a UDP version, and it worked pretty stable working

On Tue, Jun 2, 2009 at 2:20 AM, christian (via Nabble)
<ml-user+63078-1396442072 at n2.nabble.com> wrote:
> Hi,
>
> my 2 cents: The code looks ok and the approach to send the file in
> chunks and send a special "end of file" message also seems reasonable
> to me.
> The "end command" is actually sent (i.e. you see the "end chunk data"
> log msg), right?
> There is only one possibility left if no exceptions occur during the
> execution of your code (you are using tcp, right?):
>
> The server does not understand your "end command". Is the format of
> the end command you are sending really correct? Is it also your
> server, can you check if the decoding of the end message is also
> working when the srv is in "de-chunking mode"?
> If you think it is all correct you could check if the message is
> actually received on the server using Wireshark or your preffered
> network sniffer, but if no exceptions occured it should.
>
> hope this helps a little,
> regards,
> christian!
>
> On Tue, Jun 2, 2009 at 8:06 AM, Jiang Bian <timobj at ...> wrote:
>>
>> Long story short, one part of my project requires to send large files
>> using
>> netty. The large is split into small chunks, and at the end, the client is
>> suppose to send a "end command" signal. The problem is the "end command"
>> never gets to the server, when sending big files (20MB).
>>
>> Here is my code for the sendFile funciton.
>>
>>   private void sendFile(MessageEvent e) throws InterruptedException {
>>
>>        ChannelFuture lastWriteFuture = null;
>>
>>        int count = wrappedFile.getSegmentCount();
>>
>>        int sleepTime = SystemConfigHelper.getSystemConfigHelper()
>>                .getSleepTime();
>>
>>        int i = 0;
>>        while (e.getChannel().isWritable() && i < count) {
>>            for (; i < count; i++) {
>>                // logger.info("sending segment " + i);
>>                Chunk chunk = wrappedFile.getChunk(i);
>>                // logger.debug("sending: " + chunk.getMD5String());
>>                // logger.debug("length: " + chunk.getPayload().length);
>>                ChannelBuffer buffer = JigDFSProtocolRequestFactory
>>                        .chunkData(chunk);
>>
>>                this.bytesSent.addAndGet(buffer.capacity());
>>
>>                lastWriteFuture = e.getChannel().write(buffer);
>>
>>                try {
>>                    Thread.sleep(sleepTime);
>>                } catch (InterruptedException ex) {
>>                    ex.printStackTrace();
>>                    throw new InterruptedException(ex.getMessage());
>>                }
>>
>>            }
>>        }
>>
>>        // Wait until all messages are flushed before closing the channel.
>>        if (lastWriteFuture != null) {
>>            lastWriteFuture.awaitUninterruptibly();
>>        }
>>
>>        logger.debug("end chunk data");
>>
>>        logger.debug("write end_chunk_data command");
>>
>>        lastWriteFuture = e.getChannel().write(
>>                JigDFSProtocolRequestFactory.endChunkData());
>>
>>        // Wait until all messages are flushed...
>>        if (lastWriteFuture != null) {
>>            lastWriteFuture.awaitUninterruptibly();
>>        }
>>
>>        logger.info("file sent...");
>>
>>    }
>>
>> It works fine with small files. So, my question basically is: what's the
>> right way to  ensure a message gets to the server on the client side?
>> Hope I am not making stupid mistakes here.
>> Please let me know if you need any other information.
>>
>> Thanks,
>>
>> Jiang
>> --
>> View this message in context:
>> http://n2.nabble.com/Sending-large-files-tp3010547p3010547.html
>> Sent from the Netty User Group mailing list archive at Nabble.com.
>>
>> _______________________________________________
>> netty-users mailing list
>> netty-users at ...
>> https://lists.jboss.org/mailman/listinfo/netty-users
>>
> _______________________________________________
> netty-users mailing list
> netty-users at ...
> https://lists.jboss.org/mailman/listinfo/netty-users
>
>
> ________________________________
> This email is a reply to your post @
> http://n2.nabble.com/Sending-large-files-tp3010547p3010722.html
> You can reply by email or by visting the link above.
>
>

-- 
View this message in context: http://n2.nabble.com/Sending-large-files-tp3010547p3011890.html
Sent from the Netty User Group mailing list archive at Nabble.com.