[Apiman-user] API policy response data handler
Marc Savy
marc.savy at redhat.com
Fri Feb 3 06:48:00 EST 2017
Aha, you're specifically talking about the apiman UI - I had thought from
the context that you were using an external tool like Grafana.
I'll let Eric respond as he implemented that code.
On 3 February 2017 at 11:38, Balu S <sbalu27 at gmail.com> wrote:
> sorry, I don't understand what you wrote.
>
> I would like Apiman metrics to consider the HTTP response (>=400) as
> failure/error (whatever that is appropriate) and indicate in below charts.
>
>
>
> [image: Inline image 1]
>
> On Fri, Feb 3, 2017 at 12:21 PM, Marc Savy <marc.savy at redhat.com> wrote:
>
>> I think I understand what you're asking.
>>
>> IIRC:
>>
>> Error = an exception occurred within the gateway (e.g. could not connect
>> to backend).
>> Failure = a policy failure occurred.
>>
>>
>> If you're wanting to look at http exception codes, just look at the
>> 'responseCode' field in the ES Metrics.
>>
>> If you're wanting a "failed/not failed" determination then perhaps just
>> execute `responseCode/100 == 2` in your analysis? Or is this not possible
>> in your use-case?
>>
>> On 3 February 2017 at 10:45, Balu S <sbalu27 at gmail.com> wrote:
>>
>>> Its the default one ES.
>>>
>>> # ---------------------------------------------------------------------
>>> # Elasticsearch Metrics Settings
>>> # ---------------------------------------------------------------------
>>>
>>> apiman-gateway.metrics=io.apiman.gateway.engine.es.ESMetrics
>>> apiman-gateway.metrics.client.type=jest
>>> apiman-gateway.metrics.client.protocol=${apiman.es.protocol}
>>> apiman-gateway.metrics.client.host=${apiman.es.host}
>>> apiman-gateway.metrics.client.port=${apiman.es.port}
>>> apiman-gateway.metrics.client.username=${apiman.es.username}
>>> apiman-gateway.metrics.client.password=${apiman.es.password}
>>> apiman-gateway.metrics.client.timeout=${apiman.es.timeout}
>>> apiman-gateway.metrics.client.initialize=true
>>>
>>> On Fri, Feb 3, 2017 at 11:40 AM, Marc Savy <marc.savy at redhat.com> wrote:
>>>
>>>> Which metrics implementation are you using? The data you're talking
>>>> about should be in there; if it's not then there's a problem.
>>>>
>>>>
>>>> On 3 February 2017 at 09:50, Balu S <sbalu27 at gmail.com> wrote:
>>>>
>>>>> Thanks for your inputs.
>>>>>
>>>>> Yes, I mean HTTP error codes (non-200) that are returned to client.
>>>>> For example, when a request is missing query parameters and the API
>>>>> responds with a "Bad request" (400) along with error XML. Here the error
>>>>> code is set at HTTP level only and Apiman metrics should consider it as bad
>>>>> response (in my opinion). Neither I do not see the Apiman source
>>>>> (HttpApiConnection) interpreting the HTTP response using the HTTP code. But
>>>>> with the custom policy, If I check for non-200 code and handle as failure,
>>>>> then metrics shows them as error.
>>>>>
>>>>> To clarify on my implementation with custom policy, I'm not trying to
>>>>> change the HTTP error code based on the response, rather we are unpacking
>>>>> the error response body and packing in different XML format as done by
>>>>> Apiman. Is this not a valid scenario ? I think there could other scenarios
>>>>> where one want to alter the response body. I agree there will be additional
>>>>> cost to performance and memory, but can it be not done one demand basis
>>>>> like how one can implement IDataPolicy to parse the response only if he
>>>>> needs to.
>>>>>
>>>>> Thanks
>>>>> Balu
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Feb 2, 2017 at 10:31 PM, Eric Wittmann <
>>>>> eric.wittmann at redhat.com> wrote:
>>>>>
>>>>>> The bottom line here is that you cannot return a Policy Failure (or
>>>>>> customize it) based on information in the response body. The response is
>>>>>> streamed from the back-end to the client, and at the time streaming begins,
>>>>>> the response code and HTTP headers have already been sent.
>>>>>>
>>>>>> It sounds to me like you're asking for a feature where you can parse
>>>>>> the response body *before* the policy's "apply" method is invoked. We have
>>>>>> such a feature for requests, but not for responses. I suspect core changes
>>>>>> to apiman would be required to enable that. It seems like a reasonable
>>>>>> request to me, as long as users of the feature understand the performance
>>>>>> and memory requirements of enabling it.
>>>>>>
>>>>>> -Eric
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 2, 2017 at 1:03 PM, Marc Savy <marc.savy at redhat.com>
>>>>>> wrote:
>>>>>>
>>>>>>> NB: This is distinct from the body you're setting which contains a
>>>>>>> JSON/XML payload containing the error code. It's in the HTTP protocol
>>>>>>> itself.
>>>>>>>
>>>>>>> On 2 February 2017 at 18:02, Marc Savy <marc.savy at redhat.com> wrote:
>>>>>>>
>>>>>>>> That sounds like metrics are going wrong, or perhaps you're
>>>>>>>> misinterpreting it (adding Eric).
>>>>>>>>
>>>>>>>> When you say your API returns an error, does it still return an
>>>>>>>> appropriate non-200 error code at the HTTP level? For instance, 500 or
>>>>>>>> similar? That's very important.
>>>>>>>>
>>>>>>>> There's a difference between an error and a failure - have you
>>>>>>>> checked both of those fields to see whether they contain the information
>>>>>>>> you're expecting to see.
>>>>>>>>
>>>>>>>> Certainly in my experience we *do* collect the metrics you're
>>>>>>>> talking about, unless I'm misunderstanding you.
>>>>>>>>
>>>>>>>> On 2 February 2017 at 17:40, Balu S <sbalu27 at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Marc,
>>>>>>>>>
>>>>>>>>> I shall explain my use case. From our API call, we return error
>>>>>>>>> response (in XML or JSON) for 401, 500 and so on. However, Apiman metrics
>>>>>>>>> seems to just consider them as good response (it is, as Apiman received the
>>>>>>>>> response back) and show as successful response. So I have made custom
>>>>>>>>> policy to intercept the response to know if it is failure and trigger the
>>>>>>>>> Policy Failure. This is fairly simple and straight forward as the response
>>>>>>>>> code will just do the purpose. But I want also to add some additional
>>>>>>>>> information about the failure to Policy Failure. This additional
>>>>>>>>> information is in the original error response which will be lost once
>>>>>>>>> doFailure() happens. And no we don't want to those additional information
>>>>>>>>> in some HTTP headers to pass around. Hence I implemented responseHandler()
>>>>>>>>> to handle the response buffer and like you pointed out, it seems to be too
>>>>>>>>> late to meddle the response.
>>>>>>>>>
>>>>>>>>> So ideally, there are possible 2 solution
>>>>>>>>>
>>>>>>>>> - Apiman metrics can interpret the response as unsuccessful for
>>>>>>>>> such error response from API call.
>>>>>>>>> - Handle the response buffer data before the write() call to
>>>>>>>>> response outputstream.
>>>>>>>>>
>>>>>>>>> Do you see any alternative solution?
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Balu
>>>>>>>>>
>>>>>>>>> On Thu, Feb 2, 2017 at 6:15 PM, Marc Savy <marc.savy at redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Perhaps URLRewritingPolicy https://github.com/apiman/apim
>>>>>>>>>> an/blob/master/gateway/engine/policies/src/main/java/io/apim
>>>>>>>>>> an/gateway/engine/policies/URLRewritingPolicy.java be an
>>>>>>>>>> informative place to start?
>>>>>>>>>>
>>>>>>>>>> - Apiman streams data, so the client may be receiving data
>>>>>>>>>> already by the time you've determined you want to cancel (the connection is
>>>>>>>>>> already established; headers have been sent) - it's often too late to
>>>>>>>>>> gracefully cancel. You could try throwing an exception and seeing what
>>>>>>>>>> happens (not recommended practice!).
>>>>>>>>>>
>>>>>>>>>> If that doesn't work, perhaps you can explain your use-case more
>>>>>>>>>> clearly and explicitly so we can see what the alternatives are?
>>>>>>>>>>
>>>>>>>>>> - Policies are *static* instances, if you are assigning that
>>>>>>>>>> buffer to the object then it's as if you were writing "static Buffer
>>>>>>>>>> buffer" and different requests will all share that variable (and thus swap
>>>>>>>>>> it out repeatedly!).
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Marc
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 2 February 2017 at 16:56, Balu S <sbalu27 at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>> I'm trying to parse the response using responseDataHandler() in
>>>>>>>>>>> the custom policy. In cases, if the response from API is of certain
>>>>>>>>>>> content, I would like the Apiman to consider as failure. But I don't find a
>>>>>>>>>>> way to throw policy failure from responseDataHandler(). And I cannot
>>>>>>>>>>> achieve this in doApply() as the ApiResponse object does not have "content"
>>>>>>>>>>> to parse.
>>>>>>>>>>>
>>>>>>>>>>> Also, what I found is write(chunk) in the AbstractStream is
>>>>>>>>>>> called after doApply, so I cannot set any attributes in it to fetch it in
>>>>>>>>>>> doApply() and trigger doFailure().
>>>>>>>>>>>
>>>>>>>>>>> For example, in below call, how to throw as policy failure after
>>>>>>>>>>> parsing the contents ? Or how can I access response content even before
>>>>>>>>>>> write() method.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *URLRewritingPolicy.java*
>>>>>>>>>>>
>>>>>>>>>>> @Override
>>>>>>>>>>> protected IReadWriteStream<ApiResponse>
>>>>>>>>>>> responseDataHandler(ApiResponse response,
>>>>>>>>>>> IPolicyContext context, URLRewritingConfig
>>>>>>>>>>> policyConfiguration) {
>>>>>>>>>>> if (policyConfiguration.isProcessResponseBody()) {
>>>>>>>>>>> return new URLRewritingStream(context.get
>>>>>>>>>>> Component(IBufferFactoryComponent.class), response,
>>>>>>>>>>> policyConfiguration.getFromRegex(),
>>>>>>>>>>> policyConfiguration.getToReplacement());
>>>>>>>>>>> } else {
>>>>>>>>>>> return null;
>>>>>>>>>>> }
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> *URLRewritingStream.java*
>>>>>>>>>>>
>>>>>>>>>>> /**
>>>>>>>>>>> * @see io.apiman.gateway.engine.io.AbstractStream#write(io.
>>>>>>>>>>> apiman.gateway.engine.io.IApimanBuffer)
>>>>>>>>>>> */
>>>>>>>>>>> @Override
>>>>>>>>>>> public void write(IApimanBuffer chunk) {
>>>>>>>>>>> if (buffer == null) {
>>>>>>>>>>> buffer = bufferFactory.cloneBuffer(chunk);
>>>>>>>>>>> } else {
>>>>>>>>>>> buffer.append(chunk);
>>>>>>>>>>> }
>>>>>>>>>>> atEnd = false;
>>>>>>>>>>> processBuffer();
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Best regards
>>>>>>>>>>> Balu
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Apiman-user mailing list
>>>>>>>>>>> Apiman-user at lists.jboss.org
>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/apiman-user
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/apiman-user/attachments/20170203/306f9c05/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 54197 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/apiman-user/attachments/20170203/306f9c05/attachment-0001.png
More information about the Apiman-user
mailing list