HttpMessageDecoder - Skip the footer; does anyone use it?

[Roger F]

Hi Trustin Lee,

I have been trying to use TrailingHeaders after a Content-Transfer: chunked
message, obviously it does not work, yet.

This would be useful to allow large files, size unknown, to be streamed in
chunks, followed by the MD5.
This ensures data integrity and prevents out of memory in a server.

Here is an example :

PUT / HTTP/1.1
Host: localhost:8081
Transfer-Encoding: chunked
Trailer: Content-MD5

64
1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
0
Content-MD5: 4f30805221b29b36df636869549398fb



The following code changes allow TrailingHeaders and seem to work so far in
testing.

In HttpMessageDecoder replacing in the decode method the following switch
state READ_CHUNK_FOOTER :

@Override
protected Object decode(ChannelHandlerContext ctx, Channel channel,
ChannelBuffer buffer, State state) throws Exception {

....

case READ_CHUNK_FOOTER: {
    // Do not Skip the footer; some people use it!

    try {

        readFooterHeaders(buffer);

        if (maxChunkSize == 0) {
            // Chunked encoding disabled.
            return reset();
        } else {
            reset();
            // The last chunk, which is empty
            return HttpChunk.LAST_CHUNK;
        }
    } finally {
        checkpoint();
    }
}

...


Then also in HttpMessageDecoder adding the following method :

private void readFooterHeaders(ChannelBuffer buffer) throws
TooLongFrameException {
    HttpFooterHeaders footerHeaders = new HttpFooterHeaders();
    headerSize = 0;
    String line = readHeader(buffer);
    String lastHeader = null;
    if (line.length() != 0) {
        do {
            char firstChar = line.charAt(0);
            if (lastHeader != null && (firstChar == ' ' || firstChar ==
'\t')) {
                List<String> current = footerHeaders.getHeaders(lastHeader);
                int lastPos = current.size() - 1;
                String newString = current.get(lastPos) + line.trim();
                current.set(lastPos, newString);
            } else {
                String[] header = splitHeader(line);
                footerHeaders.addHeader(header[0], header[1]);
                lastHeader = header[0];
            }

            line = readHeader(buffer);
        } while (line.length() != 0);
    }

    Set<String> headerNames = footerHeaders.getHeaderNames();
    for (Iterator<String> it = headerNames.iterator(); it.hasNext();) {
        String headerName = it.next();
        List<String> headerValues = footerHeaders.getHeaders(headerName);
        for (String headerValue : headerValues) {
            message.addHeader(headerName, headerValue);
        }

    }

}


The following class needs to then be added to the package
org.jboss.netty.handler.codec.http :

package org.jboss.netty.handler.codec.http;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;

import org.jboss.netty.util.internal.CaseIgnoringComparator;

public class HttpFooterHeaders {

    private final Map<String, List<String>> headers = new TreeMap<String,
List<String>>(CaseIgnoringComparator.INSTANCE);

    public HttpFooterHeaders() {
    }

    public void addHeader(final String name, final String value) {
        validateHeaderName(name);
        validateHeaderValue(value);
        if (value == null) {
            throw new NullPointerException("value is null");
        }
        //RFC2616 section 14.40 footer headers MUST NOT contain
Transfer-Encoding, Content-Length or Trailer
        if (name.equalsIgnoreCase("Transfer-Encoding") ||
name.equalsIgnoreCase("Content-Length") || name.equalsIgnoreCase("Trailer"))
{
            return;
        }
        if (headers.get(name) == null) {
            headers.put(name, new ArrayList<String>(1));
        }
        headers.get(name).add(value);
    }

    private static void validateHeaderName(String name) {
        if (name == null) {
            throw new NullPointerException("name");
        }
        for (int i = 0; i < name.length(); i++) {
            char c = name.charAt(i);
            if (c > 127) {
                throw new IllegalArgumentException(
                        "name contains non-ascii character: " + name);
            }

            // Check prohibited characters.
            switch (c) {
                case '=':
                case ',':
                case ';':
                case ' ':
                case ':':
                case '\t':
                case '\r':
                case '\n':
                case '\f':
                case 0x0b: // Vertical tab
                    throw new IllegalArgumentException(
                            "name contains one of the following prohibited
characters: " +
                            "=,;: \\t\\r\\n\\v\\f: " + name);
            }
        }
    }

    private static void validateHeaderValue(String value) {
        if (value == null) {
            throw new NullPointerException("value");
        }
        for (int i = 0; i < value.length(); i++) {
            char c = value.charAt(i);
            // Check prohibited characters.
            switch (c) {
                case '\r':
                case '\n':
                case '\f':
                case 0x0b: // Vertical tab
                    throw new IllegalArgumentException(
                            "value contains one of the following prohibited
characters: " +
                            "\\r\\n\\v\\f: " + value);
            }
        }
    }

    public List<String> getHeaders(final String name) {
        List<String> values = headers.get(name);
        if (values == null) {
            return Collections.emptyList();
        } else {
            return values;
        }
    }

    public Set<String> getHeaderNames() {
        return headers.keySet();
    }
}


This is probably not the best way as it duplicates some of the code from
HttpMessage as the headers and methods are built into the class. 
The problem is then adding the trailing headers to the HttpMessage headers
without clearing them.
This is a quick fix, maybe you understand a better way to implement it.

I've not found the way to send chunked data from the client side, will
continue looking.

Thanks for the effort you've done so far on NIO frameworks.
We've been using Mina for a while now and it has been working very well.
We are now migrating onto Netty and so far it looks like you have done an
excellent job.


Regards
   Roger F
-- 
View this message in context: http://n2.nabble.com/HttpMessageDecoder-Skip-the-footer-does-anyone-use-it-tp3948901p3948901.html
Sent from the Netty Developer Group mailing list archive at Nabble.com.