My story on using Netty

Sat Jul 18 02:56:59 EDT 2009

Hi community,

I will introduce myself and some of my projects. It is quite long and would
probably have to be shorter if included in the success story of Netty...

I'm Frederic Bregier and I'm currently working for a government
administration IT service.
One of my projects concerns a complete solution of document archiving, from
delivery to restitution either to end users (140 000 people) or to end
business applications (more than 30 applications with totally different
business objects).

We have to face several problems that "proprietary softwares" would not be
ready to address when the project would be at full charge:
- huge amount of document injected in the system (more than 300 000 a day)
- huge amount of document retrieval from end users (more than 1 000 000 a
day)
- different sorts of retention times for the documents from some months
(50%) up to 100 years (10%), so having a lot of suppression of documents in
the system every day
- huge amount of storage with quick access (more than 1 PetaByte with
document retrieval for end users no longer than some seconds)

>From those constraints, we decided in 2006 to start several projects in open
source model:
- The very first project was to rewrite from scratch a document archiving
software that would be compatible with those constraints. At that time, MINA
was our choice for the network framework, giving the birth of the project
named OpenLSD (Legacy Storage Document). This project is not yet fully in
production, since the main software (out of the open source model) is still
in production due to contract considerations. OpenLSD is a framework for
document archiving and not a final product (some basic class have to be
implementing according to the business needs).

- From one year now, we have to face problems with a non open source
software, so we decided to make a work around on some functions on this
software (web retrieval by passing the software with other open source
peaces). When we started, we considered to use Netty between the web servers
(Tomcat) and the main data server since its API and its documentation were
already quite impressive. The performance cames too and we started to use
this. This project is not published since it is related with a non open
source software.

- From early 2009, we face up in production a problem we saw in our
qualification process but had no time at this stage to correct it. The
problem is on the proprietary network file transfer monitor that we used
(non open source) which is quite limited both in terms of network protocols
and in terms of stability when a huge number of file transfers occurs. As we
are concerned with security, reliability, performance, ..., we decided to re
start a project that was pending from 2007, named at the beginning OpenR66
(for Route 66) and which is now part of a bigger project named GoldenGate.
We decided from our previous experience in 2008 to use Netty.

GoldenGate goals are to offer a multi protocol gateway (in passive mode,
that is to say, external clients decide what and when they transfer files),
but with a business long term view for a file transfer from the gateway
reception (/or sending) up to (/from) the main data server. More precisely,
the main data server is still using first the proprietary file transfer
monitor, but will switch to OpenR66 new implementation when it will be ready
(mandatory for security reasons). The front gateway servers (Goldengate) are
using different protocols with outside clients and transforms those
protocols into the protocol supported by the main data server. The idea is
we don't want to break the business global view of the file transfer, even
if we change of network protocol in the middle.

At the time writing is here the status of this project:
- HTTP support is done right now using Tomcat native way, but we planned to
see later on (when we have more time) to try the HTTP Netty implementation.
OK

- FTP server support was completely written from scratch since we need an
FTP service that will be able to have pre or post actions on transfer
completion, which we don't find quite as we wanted into the open source
community. Of course, we wanted too to have robustness, efficiency, ... It
is an open source project (hosted in the Sourceforge GoldenGate main
project) named GoldenGateFTP.

Netty helps us quite a lot to implement this FTP server (or service,
depending on how you use it). First creating the different codec were quite
easy (one for the control command link and some for the data link). The main
difficult part was to have a relation in every case as possible with the
control link and the 
data link. After that, creating the functionallities that obey to several
RFC were quite easy. It also include some functionalities of Netty that are
right now in branch of Netty (bandwith limitation for instance).

Despite the usage we wanted to do with it, we try to create it so that
someone can use it for its own purpose (using pre or post actions) or even
as a simple FTP Server (with no actions). Perhaps it could be one day
included in Netty as a subproject, probably after some "polishing"...

The benchmarks we made on a simple dual Xeon dual core reach the following
levels:
- more than 500 connections by second on control side (with full session and
authentication).
- more than 1600 file transfers by second in passive FTP mode, and more than
1900 file transfers by second in active FTP mode (using small files to not
have bandwith limitation when testing)
- almost 90% of the bandwith (900 Mbits on a Gbits link)
Those benchmarks were made on June, and Netty is still improving its
performance so we can expect better numbers.
Netty was really stable and its community answer our questions quite fast
and precise. I would say this community is really precious for its support
as for its quality (both technical and human). Only 5 months were necessary
(for 1 developper mainly) to implement a new FTP server from scratch, I
would say mainly gracefully due to Netty.
It will enter in production in the next few months and entering too in its
official 1.0 version at the same time.

- SSH support will be made later on, and we wanted to implement it too using
Netty in order to not rely on the system level authentication (security
break). We want to implement SCP functionality for instance.

- OpenR66 (file transfer monitor) is in the process of writing from May
2009. It is the 5th times we start it from scratch since each time we don't
feel quite confident with the 4 previous attempts. Now we decided to make
profit on our knowledge and trust of Netty to implement it quite
efficiently. The network layout was made in 3 weeks (a bit more complicated
codec than FTP since it is using both network connections and local
connections in order to be able to multiplex file transfers between two
hosts on a same connection). As for now, the performance is quite
impressive, and about 60% of the process is written, business part (sgbd
data layout mainly) is what we are currently implementing.

Again, Netty makes it quite easy, surprisingly even for us that used Netty
for a while now. Trustin and all the community are really helpful when we
have to made some choice in the implementation. 

To resume, Netty has a great API, documentation and community that is
impressive, as much as the efficiency and reliability that we found in our
benchmarks or stability testing.
Our next plan would be to rewrite the network layout of the OpenLSD first
project (biggest one for us) using Netty, probably using OpenR66
implementation or so.

Thank again for all the community!
Thank to Netty!

Sincerely,
Frederic Bregier

-----
Hardware/Software Architect
-- 
View this message in context: http://n2.nabble.com/My-story-on-using-Netty-tp3279548p3279548.html
Sent from the Netty User Group mailing list archive at Nabble.com.