Re: [Apiman-user] API policy response data handler

Aha, you're specifically talking about the apiman UI - I had thought from the context that you were using an external tool like Grafana. I'll let Eric respond as he implemented that code. On 3 February 2017 at 11:38, Balu S <sbalu27(a)gmail.com&gt; wrote: > sorry, I don't understand what you wrote. > > I would like Apiman metrics to consider the HTTP response (>=400) as > failure/error (whatever that is appropriate) and indicate in below charts. > > > > [image: Inline image 1] > > On Fri, Feb 3, 2017 at 12:21 PM, Marc Savy <marc.savy(a)redhat.com&gt; wrote: > >> I think I understand what you're asking. >> >> IIRC: >> >> Error = an exception occurred within the gateway (e.g. could not connect >> to backend). >> Failure = a policy failure occurred. >> >> >> If you're wanting to look at http exception codes, just look at the >> 'responseCode' field in the ES Metrics. >> >> If you're wanting a "failed/not failed" determination then perhaps just >> execute `responseCode/100 == 2` in your analysis? Or is this not possible >> in your use-case? >> >> On 3 February 2017 at 10:45, Balu S <sbalu27(a)gmail.com&gt; wrote: >> >>> Its the default one ES. >>> >>> # --------------------------------------------------------------------- >>> # Elasticsearch Metrics Settings >>> # --------------------------------------------------------------------- >>> >>> apiman-gateway.metrics=io.apiman.gateway.engine.es.ESMetrics >>> apiman-gateway.metrics.client.type=jest >>> apiman-gateway.metrics.client.protocol=${apiman.es.protocol} >>> apiman-gateway.metrics.client.host=${apiman.es.host} >>> apiman-gateway.metrics.client.port=${apiman.es.port} >>> apiman-gateway.metrics.client.username=${apiman.es.username} >>> apiman-gateway.metrics.client.password=${apiman.es.password} >>> apiman-gateway.metrics.client.timeout=${apiman.es.timeout} >>> apiman-gateway.metrics.client.initialize=true >>> >>> On Fri, Feb 3, 2017 at 11:40 AM, Marc Savy <marc.savy(a)redhat.com&gt; >>> wrote: >>> >>>> Which metrics implementation are you using? The data you're talking >>>> about should be in there; if it's not then there's a problem. >>>> >>>> >>>> On 3 February 2017 at 09:50, Balu S <sbalu27(a)gmail.com&gt; wrote: >>>> >>>>> Thanks for your inputs. >>>>> >>>>> Yes, I mean HTTP error codes (non-200) that are returned to client. >>>>> For example, when a request is missing query parameters and the API >>>>> responds with a "Bad request" (400) along with error XML. Here the error >>>>> code is set at HTTP level only and Apiman metrics should consider it as bad >>>>> response (in my opinion). Neither I do not see the Apiman source >>>>> (HttpApiConnection) interpreting the HTTP response using the HTTP code. But >>>>> with the custom policy, If I check for non-200 code and handle as failure, >>>>> then metrics shows them as error. >>>>> >>>>> To clarify on my implementation with custom policy, I'm not trying to >>>>> change the HTTP error code based on the response, rather we are unpacking >>>>> the error response body and packing in different XML format as done by >>>>> Apiman. Is this not a valid scenario ? I think there could other scenarios >>>>> where one want to alter the response body. I agree there will be additional >>>>> cost to performance and memory, but can it be not done one demand basis >>>>> like how one can implement IDataPolicy to parse the response only if he >>>>> needs to. >>>>> >>>>> Thanks >>>>> Balu >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, Feb 2, 2017 at 10:31 PM, Eric Wittmann < >>>>> eric.wittmann(a)redhat.com&gt; wrote: >>>>> >>>>>> The bottom line here is that you cannot return a Policy Failure (or >>>>>> customize it) based on information in the response body. The response is >>>>>> streamed from the back-end to the client, and at the time streaming begins, >>>>>> the response code and HTTP headers have already been sent. >>>>>> >>>>>> It sounds to me like you're asking for a feature where you can parse >>>>>> the response body *before* the policy's "apply" method is invoked. We have >>>>>> such a feature for requests, but not for responses. I suspect core changes >>>>>> to apiman would be required to enable that. It seems like a reasonable >>>>>> request to me, as long as users of the feature understand the performance >>>>>> and memory requirements of enabling it. >>>>>> >>>>>> -Eric >>>>>> >>>>>> >>>>>> On Thu, Feb 2, 2017 at 1:03 PM, Marc Savy <marc.savy(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> NB: This is distinct from the body you're setting which contains a >>>>>>> JSON/XML payload containing the error code. It's in the HTTP protocol >>>>>>> itself. >>>>>>> >>>>>>> On 2 February 2017 at 18:02, Marc Savy <marc.savy(a)redhat.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> That sounds like metrics are going wrong, or perhaps you're >>>>>>>> misinterpreting it (adding Eric). >>>>>>>> >>>>>>>> When you say your API returns an error, does it still return an >>>>>>>> appropriate non-200 error code at the HTTP level? For instance, 500 or >>>>>>>> similar? That's very important. >>>>>>>> >>>>>>>> There's a difference between an error and a failure - have you >>>>>>>> checked both of those fields to see whether they contain the information >>>>>>>> you're expecting to see. >>>>>>>> >>>>>>>> Certainly in my experience we *do* collect the metrics you're >>>>>>>> talking about, unless I'm misunderstanding you. >>>>>>>> >>>>>>>> On 2 February 2017 at 17:40, Balu S <sbalu27(a)gmail.com&gt; wrote: >>>>>>>> >>>>>>>>> Hi Marc, >>>>>>>>> >>>>>>>>> I shall explain my use case. From our API call, we return error >>>>>>>>> response (in XML or JSON) for 401, 500 and so on. However, Apiman metrics >>>>>>>>> seems to just consider them as good response (it is, as Apiman received the >>>>>>>>> response back) and show as successful response. So I have made custom >>>>>>>>> policy to intercept the response to know if it is failure and trigger the >>>>>>>>> Policy Failure. This is fairly simple and straight forward as the response >>>>>>>>> code will just do the purpose. But I want also to add some additional >>>>>>>>> information about the failure to Policy Failure. This additional >>>>>>>>> information is in the original error response which will be lost once >>>>>>>>> doFailure() happens. And no we don't want to those additional information >>>>>>>>> in some HTTP headers to pass around. Hence I implemented responseHandler() >>>>>>>>> to handle the response buffer and like you pointed out, it seems to be too >>>>>>>>> late to meddle the response. >>>>>>>>> >>>>>>>>> So ideally, there are possible 2 solution >>>>>>>>> >>>>>>>>> - Apiman metrics can interpret the response as unsuccessful >>>>>>>>> for such error response from API call. >>>>>>>>> - Handle the response buffer data before the write() call to >>>>>>>>> response outputstream. >>>>>>>>> >>>>>>>>> Do you see any alternative solution? >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Balu >>>>>>>>> >>>>>>>>> On Thu, Feb 2, 2017 at 6:15 PM, Marc Savy <marc.savy(a)redhat.com&gt; >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> Perhaps URLRewritingPolicy https://github.com/apiman/apim >>>>>>>>>> an/blob/master/gateway/engine/policies/src/main/java/io/apim >>>>>>>>>> an/gateway/engine/policies/URLRewritingPolicy.java be an >>>>>>>>>> informative place to start? >>>>>>>>>> >>>>>>>>>> - Apiman streams data, so the client may be receiving data >>>>>>>>>> already by the time you've determined you want to cancel (the connection is >>>>>>>>>> already established; headers have been sent) - it's often too late to >>>>>>>>>> gracefully cancel. You could try throwing an exception and seeing what >>>>>>>>>> happens (not recommended practice!). >>>>>>>>>> >>>>>>>>>> If that doesn't work, perhaps you can explain your use-case more >>>>>>>>>> clearly and explicitly so we can see what the alternatives are? >>>>>>>>>> >>>>>>>>>> - Policies are *static* instances, if you are assigning that >>>>>>>>>> buffer to the object then it's as if you were writing "static Buffer >>>>>>>>>> buffer" and different requests will all share that variable (and thus swap >>>>>>>>>> it out repeatedly!). >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Marc >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 2 February 2017 at 16:56, Balu S <sbalu27(a)gmail.com&gt; wrote: >>>>>>>>>> >>>>>>>>>>> Hello, >>>>>>>>>>> I'm trying to parse the response using responseDataHandler() in >>>>>>>>>>> the custom policy. In cases, if the response from API is of certain >>>>>>>>>>> content, I would like the Apiman to consider as failure. But I don't find a >>>>>>>>>>> way to throw policy failure from responseDataHandler(). And I cannot >>>>>>>>>>> achieve this in doApply() as the ApiResponse object does not have "content" >>>>>>>>>>> to parse. >>>>>>>>>>> >>>>>>>>>>> Also, what I found is write(chunk) in the AbstractStream is >>>>>>>>>>> called after doApply, so I cannot set any attributes in it to fetch it in >>>>>>>>>>> doApply() and trigger doFailure(). >>>>>>>>>>> >>>>>>>>>>> For example, in below call, how to throw as policy failure >>>>>>>>>>> after parsing the contents ? Or how can I access response content even >>>>>>>>>>> before write() method. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *URLRewritingPolicy.java* >>>>>>>>>>> >>>>>>>>>>> @Override >>>>>>>>>>> protected IReadWriteStream<ApiResponse> >>>>>>>>>>> responseDataHandler(ApiResponse response, >>>>>>>>>>> IPolicyContext context, URLRewritingConfig >>>>>>>>>>> policyConfiguration) { >>>>>>>>>>> if (policyConfiguration.isProcessResponseBody()) { >>>>>>>>>>> return new URLRewritingStream(context.get >>>>>>>>>>> Component(IBufferFactoryComponent.class), response, >>>>>>>>>>> policyConfiguration.getFromRegex(), >>>>>>>>>>> policyConfiguration.getToReplacement()); >>>>>>>>>>> } else { >>>>>>>>>>> return null; >>>>>>>>>>> } >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> *URLRewritingStream.java* >>>>>>>>>>> >>>>>>>>>>> /** >>>>>>>>>>> * @see io.apiman.gateway.engine.io.Ab >>>>>>>>>>> stractStream#write(io.apiman.gateway.engine.io.IApimanBuffer) >>>>>>>>>>> */ >>>>>>>>>>> @Override >>>>>>>>>>> public void write(IApimanBuffer chunk) { >>>>>>>>>>> if (buffer == null) { >>>>>>>>>>> buffer = bufferFactory.cloneBuffer(chunk); >>>>>>>>>>> } else { >>>>>>>>>>> buffer.append(chunk); >>>>>>>>>>> } >>>>>>>>>>> atEnd = false; >>>>>>>>>>> processBuffer(); >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Best regards >>>>>>>>>>> Balu >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Apiman-user mailing list >>>>>>>>>>> Apiman-user(a)lists.jboss.org >>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/apiman-user >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >