[undertow-dev] Lost requests when responding from Akka actor system

[Stuart Douglas]

It is possible there may be a thread safety issue. When you do the initial dispatch try using the version of dispatch that takes a Runnable and dispatch to Akka in the Runnable.

This makes sure that there is only one thread active on the exchange at any time (as Undertow needs to examine some state on the exchange when the call stack initially returns).

Stuart

----- Original Message -----
> From: "Michael Barton" <mbarton at itrsgroup.com>
> To: undertow-dev at lists.jboss.org
> Sent: Friday, 6 March, 2015 9:06:54 PM
> Subject: [undertow-dev] Lost requests when responding from Akka actor system
> 
> 
> 
> Hi all,
> 
> 
> 
> I’ve been using Undertow embedded in an Akka application and it is working
> great apart from a small issue when running load tests.
> 
> 
> 
> I’ve reduced the problem down to a small little demo application which I’ve
> attached below. It is in Scala but I can create a Java version if required.
> 
> 
> 
> I fire one million POST requests at Undertow and almost all succeed, however
> some never receive a response. The number that fail varies from 1 to 10.
> 
> 
> 
> Increasing the concurrency on the client and server side together and
> independently increases the number of failures. I’ve not yet seen any
> failures with no concurrency on the server side.
> 
> 
> 
> The HTTP handler callback is not invoked for the requests that fail. I’ve
> verified this by adding a counter before I call exchange.dispatch() and pass
> it on to the processing actor pool.
> 
> 
> 
> I don’t think I’m doing anything that is not thread-safe since Akka
> guarantees that actors do not execute concurrently. However the execution of
> the actor does move around a number of threads in the underlying Fork Join
> pool and I’m wondering if that would cause issues with Undertow?
> 
> 
> 
> Any suggestions for how I could debug this?
> 
> 
> 
> Thanks,
> 
> 
> 
> Michael
> 
> 
> 
> Code to reproduce
> 
> ==============
> 
> 
> 
> object UndertowTest {
> 
> def start(handler: HttpHandler) = Undertow.builder()
> 
> .addHttpListener(8888, "localhost")
> 
> .setHandler(handler)
> 
> .build()
> 
> .start()
> 
> }
> 
> 
> 
> class Responder extends HttpHandler {
> 
> override def handleRequest(exchange: HttpServerExchange): Unit = {
> 
> exchange.getResponseHeaders.put(io.undertow.util.Headers.CONTENT_TYPE,
> "text/plain")
> 
> exchange.getResponseSender.send("Hello world!")
> 
> }
> 
> }
> 
> 
> 
> class RequestHandler extends Actor with Loggable {
> 
> val handler = new Responder
> 
> 
> 
> override def receive = {
> 
> case exchange: HttpServerExchange =>
> 
> handler.handeRequest(exchange)
> 
> }
> 
> }
> 
> 
> 
> object UndertowAkkaTest extends App {
> 
> val sys = ActorSystem()
> 
> 
> 
> // Changing the number here changes the number of actors handling incoming
> requests
> 
> val handlerPool =
> sys.actorOf(RoundRobinPool(4).props(Props(classOf[RequestHandler])))
> 
> 
> 
> UndertowTest.start(new HttpHandler {
> 
> override def handleRequest(exchange: HttpServerExchange): Unit = {
> 
> exchange.dispatch()
> 
> handlerPool ! exchange
> 
> }
> 
> })
> 
> }
> 
> 
> 
> Apache Bench command to test
> 
> ========================
> 
> 
> 
> # -c changes number of concurrent client side requests
> 
> ab -k -c 4 -n 100000 -p payload.json -T application/json
> http://localhost:8888/streams/demo/infrastructure/cpu
> 
> 
> 
> Data in payload.json
> 
> ===============
> 
> 
> 
> {
> 
> "sampleTime": "2015-02-24T06:02:47",
> 
> "contributor": "3b8da322-ef8f-4f6b-93a3-a171dd794308",
> 
> "host": "some-host",
> 
> "process": "some-process",
> 
> "user": 35.7,
> 
> "kernel": 12.3
> 
> }
> 
> 
> 
> 
> 
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________
> 
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev