Peregrine - a Map Reduce framework using Netty as an HTTP framework.

[Kevin Burton]

Hey guys.

This is what I've been using Netty with:

https://bitbucket.org/burtonator/peregrine/wiki/Home

It's still alpha now and is just now running 10GB jobs.

Initially, I decided to just use Netty for the HTTP framework but
ChannelBuffer sucked me in and I'm probably going to have all of our IO
driven by ChannelBuffers...

There are just a bunch of performance benefits to doing it this way.

I also ended up building out some support for extending Map Reduce so that
Keys and Values are now structs.  This way you can do things like
value.writeInt, value.writeDouble, etc.  I will probably end up having these
be facades on top of ChannelBuffer.

Hadoop uses Jetty for its HTTP support which means there's one thread per
client.  Some design decisions in Hadoop were pinned based on this and they
weren't able to do some clever hacks that I can now do in Peregrine.

Kevin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/netty-users/attachments/20111027/5ba1f34d/attachment.html