How to properly handle decoder chain in netty?

[Frederic Bregier]

Hi Jiang,

I think that corrupted data on huge transfer is probably normal.

My view is based on the following.
When you want to download a huge file (say a ISO file of your favorite
linux distribution, about 4 GB), you have often three pointers:
- one link to an HTTP download link with a "warning" that says "you might
better want to use the FTP protocol since HTTP download can corrupt data" 
- one link to an FTP download link
- one link to a MD5 file with a "warning" saying "for such huge file, you
should verify the correctness of your download with the following MD5 key"

So based on this example, my guess is that sending huge file (by chunk or
not) can lead to some bad transfer that TCP/IP (or even more UDP) cannot
address.
That's why most of the "financial" transfer file software integrate a
checksum during the transfer to validate each file (or even each chunk).

My suggestion could be the following (but take it only as a suggestion), you
could perhaps integrate something like a checksum in your protocol, in two
ways:
- either at the end of the transfer, you send the checksum (MD5 for
instance) and compares it on the target host (each checksum computed
probably at both side during the transfer in order to prevent a new read of
the file)
- either for each chunk, you send too a checksum of this chunk to be
compared on the remote host

You can even include a count number that orders the chunks (for instance to
see if there is an error in the transmission where a chunk number 1024
occurs after a chunk number 1021, meaning the chunks with numbers 1022 and
1023 are missing and to enable the restart from chunk 1022).

But don't go too fast, because including such behaviour in your protocol
will increase greatly the computation (checksum, order of chunk, retry, ...)
and so decrease the efficiency. In such problem, it is always a choice
between efficiency and security, depending on your final goals...

HTH,
Frederic


-----
Hardware/Software Architect
-- 
View this message in context: http://n2.nabble.com/How-to-properly-handle-decoder-chain-in-netty--tp3015408p3018693.html
Sent from the Netty User Group mailing list archive at Nabble.com.