Hi,
I packed all buzzwords into the subject and I promise to talk about real issues from now
on.
This is about issue
https://issues.jboss.org/browse/UNDERTOW-626, the background, possible
solutions and workarounds. But first I’d like to thank Steward for his quick response to
the issue and also to my problem with HotSpot 1.8.0_71. Build problem solved!
The Background
I live in a country that has 9 states. In the area I work, they share a datacenter. Some
applications are organised centrally (one database for all states), some are organised
locally (each state has its own database and its own application server). We have a rather
big application that is organised locally. Since there are quite a few users and we want
to have good availability, each state has at least two JBoss servers (EAP6.4 right now).
So we have about 20 servers in production. The problem now is, that they don’t live an
isolated life, but sometimes they have to talk to each other. And not only with each other
but also with a few other applications, some of them organised locally, some centrally.
Usually they communicate synchronously with web services. Usually web services are
configured point-to-point. This creates quite some configuration challenge. And I’m not
talking about micro services here.
To ease the problem, a kind of hub and spoke architecture was introduced. Apache httpd
with mod_jk is put in front of some applications. This eases configuration to some degree.
But it has its downsides. When you kill the hub, all lights go out. Been there, done that,
not fun! Also when you add some servers or applications, you have to configure the central
mod_jk. The later problem can be solved with mod_cluster. I love mod_cluster! You don’t
have to configure it. Just add servers, deploy or undeploy applications on JBoss, no
configuration on the central hub necessary.
Architects recommend a bus-architecture to overcome the downsides of a central hub. A
service bus should have a decentral organisation similar to a hardware bus. Applications
connect to the bus and should be able to exchange messages without further configuration.
However, when I looked at products called “Enterprise Service Bus”, I had difficulties to
see the bus, most look like hub and spoke architectures.
The Idea
But then the Undertow team announced the support for mod_cluster in Undertow, which got me
really excited. Now it could be possible to create a service bus to connect WildFly
servers. You just configure mod_cluster within WildFly and you send web service requests
to your own server (localhost) instead of the real destination
(
https://developer.jboss.org/thread/249311). Because mod_cluster knows about the location
of all other servers with the same multicast-address (the bus-adress), it sends the
request to the final destination. A real service bus, almost configuration free and comes
out of the box with WildFly. Is it only me or do others think as well that this could be a
real burner?
The Proof of Concept
Yes, it does work! We created a setup with two nodes, which shows that the idea can work:
https://github.com/AdrianFarmadin/modcluster-invmhttp-example
Unfortunately we ran into a problem, which I was not able to solve properly until now.
The Problem
The problem is described in
https://issues.jboss.org/browse/UNDERTOW-626. When you have
local context and modcluster filter, modcluster sends requests first to other servers and
creates a loop.
The Solutions or Workarounds
Excluding certain contexts
In UNDERTOW-626 Steward suggests:
<filter-ref name="modcluster" predicate="not path-prefix(/myapp)”/>
Unfortunately i couldn’t test it until now, but I will soon. It could work for our special
case, since we have only one application on one server and only a few contexts. Something
like that would have to work:
<filter-ref name="modcluster" predicate="not path-prefix(/myapp) and not
path-prefix(/myappws) and not path-prefix(/myappadmin)”/>
Additionally we don’t deploy or undeploy applications at runtime, so we don’t have to
configure this dynamically.
The solution will be problematic, if you e.g. undeploy one out of several applications on
WildFly. Then you would have to reconfigure the predicate and remove some parts of the
"and not path-prefix(/…)” This makes it rather difficult.
In modcluster filter: Don’t handle context, if its from the local server
In
https://github.com/undertow-io/undertow/pull/356/files we tried to change the mod
cluster filter such that it doesn’t forward a request if it the context exists on the
local server.
We used this for our proof of concept for which it worked.
The problem is, that is uses conventions from WildFly to answer the question “What is my
local jvmroute”. The system-property “jboss.node.name” is the default in WildFly, but
thats not necessarily true otherwise. But we couldn’t find how to access the configuration
property instance-id within undertow.
Fix it in WildFly
In
https://github.com/undertow-io/undertow/pull/356 Steward pointed out that the problem
should be solved in WildFly. Could you point us in a direction where we could go? We are
willing to invest some time into this issue, but we don’t have enough inside knowledge
from Undertow or WildFly jet.
Put modcluster at the end of the handler chain
Right now modcluster is invoked before the deployed applications are invoked. Maybe one
could do it the other way round and invoke modcluster only if the deployed applications
return “not found”? I just don’t know enough about Undertow to decide whether this is
feasible or just a stupid idea.
Other?
The End
Any remarks, ideas, and help are welcome.
Erhard