[undertow-dev] Lost requests when responding from Akka actor system

[Michael Barton]

Thanks Stuart.

Looks like that has fixed  the issue.

Michael

-----Original Message-----
From: Stuart Douglas [mailto:sdouglas at redhat.com] 
Sent: 07 March 2015 05:01
To: Michael Barton
Cc: undertow-dev at lists.jboss.org
Subject: Re: [undertow-dev] Lost requests when responding from Akka actor system

It is possible there may be a thread safety issue. When you do the initial dispatch try using the version of dispatch that takes a Runnable and dispatch to Akka in the Runnable.

This makes sure that there is only one thread active on the exchange at any time (as Undertow needs to examine some state on the exchange when the call stack initially returns).

Stuart

----- Original Message -----
> From: "Michael Barton" <mbarton at itrsgroup.com>
> To: undertow-dev at lists.jboss.org
> Sent: Friday, 6 March, 2015 9:06:54 PM
> Subject: [undertow-dev] Lost requests when responding from Akka actor 
> system
> 
> 
> 
> Hi all,
> 
> 
> 
> I’ve been using Undertow embedded in an Akka application and it is 
> working great apart from a small issue when running load tests.
> 
> 
> 
> I’ve reduced the problem down to a small little demo application which 
> I’ve attached below. It is in Scala but I can create a Java version if required.
> 
> 
> 
> I fire one million POST requests at Undertow and almost all succeed, 
> however some never receive a response. The number that fail varies from 1 to 10.
> 
> 
> 
> Increasing the concurrency on the client and server side together and 
> independently increases the number of failures. I’ve not yet seen any 
> failures with no concurrency on the server side.
> 
> 
> 
> The HTTP handler callback is not invoked for the requests that fail. 
> I’ve verified this by adding a counter before I call 
> exchange.dispatch() and pass it on to the processing actor pool.
> 
> 
> 
> I don’t think I’m doing anything that is not thread-safe since Akka 
> guarantees that actors do not execute concurrently. However the 
> execution of the actor does move around a number of threads in the 
> underlying Fork Join pool and I’m wondering if that would cause issues with Undertow?
> 
> 
> 
> Any suggestions for how I could debug this?
> 
> 
> 
> Thanks,
> 
> 
> 
> Michael
> 
> 
> 
> Code to reproduce
> 
> ==============
> 
> 
> 
> object UndertowTest {
> 
> def start(handler: HttpHandler) = Undertow.builder()
> 
> .addHttpListener(8888, "localhost")
> 
> .setHandler(handler)
> 
> .build()
> 
> .start()
> 
> }
> 
> 
> 
> class Responder extends HttpHandler {
> 
> override def handleRequest(exchange: HttpServerExchange): Unit = {
> 
> exchange.getResponseHeaders.put(io.undertow.util.Headers.CONTENT_TYPE,
> "text/plain")
> 
> exchange.getResponseSender.send("Hello world!")
> 
> }
> 
> }
> 
> 
> 
> class RequestHandler extends Actor with Loggable {
> 
> val handler = new Responder
> 
> 
> 
> override def receive = {
> 
> case exchange: HttpServerExchange =>
> 
> handler.handeRequest(exchange)
> 
> }
> 
> }
> 
> 
> 
> object UndertowAkkaTest extends App {
> 
> val sys = ActorSystem()
> 
> 
> 
> // Changing the number here changes the number of actors handling 
> incoming requests
> 
> val handlerPool =
> sys.actorOf(RoundRobinPool(4).props(Props(classOf[RequestHandler])))
> 
> 
> 
> UndertowTest.start(new HttpHandler {
> 
> override def handleRequest(exchange: HttpServerExchange): Unit = {
> 
> exchange.dispatch()
> 
> handlerPool ! exchange
> 
> }
> 
> })
> 
> }
> 
> 
> 
> Apache Bench command to test
> 
> ========================
> 
> 
> 
> # -c changes number of concurrent client side requests
> 
> ab -k -c 4 -n 100000 -p payload.json -T application/json 
> http://localhost:8888/streams/demo/infrastructure/cpu
> 
> 
> 
> Data in payload.json
> 
> ===============
> 
> 
> 
> {
> 
> "sampleTime": "2015-02-24T06:02:47",
> 
> "contributor": "3b8da322-ef8f-4f6b-93a3-a171dd794308",
> 
> "host": "some-host",
> 
> "process": "some-process",
> 
> "user": 35.7,
> 
> "kernel": 12.3
> 
> }
> 
> 
> 
> 
> 
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com 
> ______________________________________________________________________
> 
> _______________________________________________
> undertow-dev mailing list
> undertow-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/undertow-dev

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com ______________________________________________________________________

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________