[Apiman-user] Stream closed / Broken Pipe issues with custom plugin
Marc Savy
marc.savy at redhat.com
Fri Jan 6 09:16:48 EST 2017
Hi Harry & David,
Very interesting and glad to see a workaround/solution.
I wonder if it's a manifestation of this bug:
https://issues.jboss.org/browse/UNDERTOW-849
In the servlet implementation we use Undertow. I'll do some more research
to see whether we can handle this more gracefully and in a more informative
manner.
May I ask what sort of load levels you were seeing that made the errors
evident?
Regards,
Marc
On 6 January 2017 at 03:51, Harry Trinta <harrytpc at gmail.com> wrote:
> Hi, David!
>
> I had exactly this problem. It is not relationed with the Apiman. The
> problem is with Wildfly. If your Wildfly is with default configuration, you
> have a limit of concurrent requests (Default "*Task max threads" = *16 *
> vCPUs). So, when the Wildfly is working in the limit, it create a queue of
> requests.
>
>
> Suggested settings for your wildfly (adjust according to your server):
>
>
> Configuration -> Subsystem -> IO
>
> WORKER
> Io threads: 128
> Task max threads: 2048
> Stack size: 0
> Task keepalive: 120
>
> BUFFER POLL
> Buffer size: 16384
> Buffers per slice: 128
> Direct buffers: true
>
> 2017-01-05 17:54 GMT-02:00 David Rush <David.Rush at ihsmarkit.com>:
>
>> Good afternoon,
>>
>>
>>
>> I’m having issues with on a production API which has started to receive
>> high volumes of traffic. At peak volume times I see many exceptions being
>> thrown in the logs. The stack is indicating that an error occurred in the
>> policy chain, but when it tries to write the error to the response the
>> connection has been closed. There seems to be a couple of flavors of
>> IOException (broken pipe or stream closed from undertow). My plugin is
>> calling chain.doApply(request) when it succeeds but the stack trace is
>> indicating there is then an exception being caught in
>> doApply(Chain.java:153). I am on version 1.2.2-Final.
>>
>>
>>
>> 2017-01-05 18:28:49,095 ERROR [stderr] (default task-17)
>> java.io.IOException: UT010029: Stream is closed
>>
>> 2017-01-05 18:28:49,095 ERROR [stderr] (default task-17) at
>> io.undertow.servlet.spec.ServletOutputStreamImpl.write(Servl
>> etOutputStreamImpl.java:136)
>>
>> 2017-01-05 18:28:49,095 ERROR [stderr] (default task-17) at
>> io.undertow.servlet.spec.ServletOutputStreamImpl.write(Servl
>> etOutputStreamImpl.java:128)
>>
>> 2017-01-05 18:28:49,095 ERROR [stderr] (default task-17) at
>> io.apiman.gateway.platforms.servlet.GatewayServlet$4.write(
>> GatewayServlet.java:406)
>>
>> 2017-01-05 18:28:49,095 ERROR [stderr] (default task-17) at
>> io.apiman.gateway.platforms.servlet.GatewayServlet$4.write(
>> GatewayServlet.java:395)
>>
>> 2017-01-05 18:28:49,095 ERROR [stderr] (default task-17) at
>> io.apiman.gateway.engine.impl.DefaultPolicyErrorWriter.write
>> (DefaultPolicyErrorWriter.java:87)
>>
>> 2017-01-05 18:28:49,095 ERROR [stderr] (default task-17) at
>> io.apiman.gateway.platforms.servlet.GatewayServlet.writeErro
>> r(GatewayServlet.java:392)
>>
>> 2017-01-05 18:28:49,095 ERROR [stderr] (default task-17) at
>> io.apiman.gateway.platforms.servlet.GatewayServlet$1.handle(
>> GatewayServlet.java:210)
>>
>> 2017-01-05 18:28:49,095 ERROR [stderr] (default task-17) at
>> io.apiman.gateway.platforms.servlet.GatewayServlet$1.handle(
>> GatewayServlet.java:157)
>>
>> 2017-01-05 18:28:49,095 ERROR [stderr] (default task-17) at
>> io.apiman.gateway.engine.impl.ApiRequestExecutorImpl.lambda$
>> wrapResultHandler$0(ApiRequestExecutorImpl.java:159)
>>
>> 2017-01-05 18:28:49,095 ERROR [stderr] (default task-17) at
>> io.apiman.gateway.engine.impl.ApiRequestExecutorImpl.lambda$
>> createPolicyErrorHandler$17(ApiRequestExecutorImpl.java:614)
>>
>> 2017-01-05 18:28:49,095 ERROR [stderr] (default task-17) at
>> io.apiman.gateway.engine.policy.Chain.throwError(Chain.java:249)
>>
>> 2017-01-05 18:28:49,095 ERROR [stderr] (default task-17) at
>> io.apiman.gateway.engine.policy.Chain.doApply(Chain.java:153)
>>
>> 2017-01-05 18:28:49,096 ERROR [stderr] (default task-17) at
>> com.markit.ondemand.auth.apiman.plugins.pingfed_oauth_policy
>> .PingFedOauthPolicy.doSuccess(PingFedOauthPolicy.java:114)
>>
>> 2017-01-05 18:28:49,096 ERROR [stderr] (default task-17) at
>> com.markit.ondemand.auth.apiman.plugins.pingfed_oauth_policy
>> .PingFedOauthPolicy.access$100(PingFedOauthPolicy.java:38)
>>
>> 2017-01-05 18:28:49,096 ERROR [stderr] (default task-17) at
>> com.markit.ondemand.auth.apiman.plugins.pingfed_oauth_policy
>> .PingFedOauthPolicy$2.handle(PingFedOauthPolicy.java:193)
>>
>> 2017-01-05 18:28:49,096 ERROR [stderr] (default task-17) at
>> com.markit.ondemand.auth.apiman.plugins.pingfed_oauth_policy
>> .PingFedOauthPolicy$2.handle(PingFedOauthPolicy.java:174)
>>
>> 2017-01-05 18:28:49,096 ERROR [stderr] (default task-17) at
>> io.apiman.gateway.platforms.servlet.components.HttpClientReq
>> uestImpl.end(HttpClientRequestImpl.java:140)
>>
>> 2017-01-05 18:28:49,096 ERROR [stderr] (default task-17) at
>> com.markit.ondemand.auth.apiman.plugins.pingfed_oauth_policy
>> .PingFedOauthPolicy.retrieveAccessTokenFromPing(PingFedOauth
>> Policy.java:232)
>>
>> 2017-01-05 18:28:49,096 ERROR [stderr] (default task-17) at
>> com.markit.ondemand.auth.apiman.plugins.pingfed_oauth_policy
>> .PingFedOauthPolicy.retrieveAccessTokenFromPing(PingFedOauth
>> Policy.java:174)
>>
>> 2017-01-05 18:28:49,096 ERROR [stderr] (default task-17) at
>> com.markit.ondemand.auth.apiman.plugins.pingfed_oauth_policy
>> .PingFedOauthPolicy.doApply(PingFedOauthPolicy.java:83)
>>
>> 2017-01-05 18:28:49,096 ERROR [stderr] (default task-17) at
>> com.markit.ondemand.auth.apiman.plugins.pingfed_oauth_policy
>> .PingFedOauthPolicy.doApply(PingFedOauthPolicy.java:38)
>>
>>
>>
>> My plugin is making either an external request to ElasticSearch or an
>> HTTP request using IHttpClientComponent and then calling chain.doApply or
>> chain.doFailure from within the IAsyncResultHandler handle method. Is
>> there a problem with that pattern?
>>
>>
>>
>> Any help you can provide would be great.
>>
>> Thanks,
>>
>>
>>
>> [image: IHS Markit] <https://ihsmarkit.com/>
>>
>> * David Rush*
>> Director, API Development | Markit Digital
>> 5775 Flatiron Parkway | Boulder, CO 80301
>> *P:* +1 303 583 4244 <(303)%20583-4244> *Main:* +1 303 417 9999
>> <(303)%20417-9999>
>> david.rush at ihsmarkit.com
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------
>>
>> This e-mail, including accompanying communications and attachments, is
>> strictly confidential and only for the intended recipient. Any retention,
>> use or disclosure not expressly authorised by Markit is prohibited. This
>> email is subject to all waivers and other terms at the following link:
>> http://www.markit.com/en/about/legal/email-disclaimer.page
>>
>> Please visit http://www.markit.com/en/about/contact/contact-us.page for
>> contact information on our offices worldwide.
>>
>> _______________________________________________
>> Apiman-user mailing list
>> Apiman-user at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/apiman-user
>>
>>
>
> _______________________________________________
> Apiman-user mailing list
> Apiman-user at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/apiman-user
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/apiman-user/attachments/20170106/bb6b8658/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 6881 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/apiman-user/attachments/20170106/bb6b8658/attachment-0001.png
More information about the Apiman-user
mailing list