[Apiman-user] API policy response data handler

Balu S sbalu27 at gmail.com
Sat Feb 4 15:00:40 EST 2017


Hi Eric,
yes that looks promising solution for us. I shall try and come back.Thanks

regards
Balu

On Fri, Feb 3, 2017 at 10:31 PM, Eric Wittmann <eric.wittmann at redhat.com>
wrote:

> So I think at least one aspect of this is solvable.  You can modify the
> request metric data directly in a custom policy.  You can access and modify
> the request metric like this:
>
> {code}
>     protected void doApply(ApiResponse response, IPolicyContext context, C
> config, IPolicyChain<ApiResponse> chain) {
>      RequestMetric metric = context.getAttribute("apiman.request-metric",
> null);
>     metric.setFailure(true);
>     metric.setFailureCode(12345);
>     metric.setFailureReason("Backend API failed for a reason!"";
>         chain.doApply(response);
>     }
> {code}
>
> That will allow you to make sure the metrics information is correct for
> your use-case.  The policy context (and thus the request metric object) is
> also available to data policies:
>
> https://github.com/apiman/apiman/blob/master/gateway/
> engine/core/src/main/java/io/apiman/gateway/engine/policy/
> IDataPolicy.java#L53-L54
>
> Does that make sense?  Is there more to your use-case?
>
> -Eric
>
>
> On Fri, Feb 3, 2017 at 6:48 AM, Marc Savy <marc.savy at redhat.com> wrote:
>
>> Aha, you're specifically talking about the apiman UI - I had thought from
>> the context that you were using an external tool like Grafana.
>>
>> I'll let Eric respond as he implemented that code.
>>
>> On 3 February 2017 at 11:38, Balu S <sbalu27 at gmail.com> wrote:
>>
>>> sorry, I don't understand what you wrote.
>>>
>>> I would like Apiman metrics to consider the HTTP response (>=400) as
>>> failure/error (whatever that is appropriate) and indicate in below charts.
>>>
>>>
>>>
>>> [image: Inline image 1]
>>>
>>> On Fri, Feb 3, 2017 at 12:21 PM, Marc Savy <marc.savy at redhat.com> wrote:
>>>
>>>> I think I understand what you're asking.
>>>>
>>>> IIRC:
>>>>
>>>> Error = an exception occurred within the gateway (e.g. could not
>>>> connect to backend).
>>>> Failure = a policy failure occurred.
>>>>
>>>>
>>>> If you're wanting to look at http exception codes, just look at the
>>>> 'responseCode' field in the ES Metrics.
>>>>
>>>> If you're wanting a "failed/not failed" determination then perhaps just
>>>> execute `responseCode/100 == 2` in your analysis? Or is this not possible
>>>> in your use-case?
>>>>
>>>> On 3 February 2017 at 10:45, Balu S <sbalu27 at gmail.com> wrote:
>>>>
>>>>> Its the default one ES.
>>>>>
>>>>> # ------------------------------------------------------------
>>>>> ---------
>>>>> # Elasticsearch Metrics Settings
>>>>> # ------------------------------------------------------------
>>>>> ---------
>>>>>
>>>>> apiman-gateway.metrics=io.apiman.gateway.engine.es.ESMetrics
>>>>> apiman-gateway.metrics.client.type=jest
>>>>> apiman-gateway.metrics.client.protocol=${apiman.es.protocol}
>>>>> apiman-gateway.metrics.client.host=${apiman.es.host}
>>>>> apiman-gateway.metrics.client.port=${apiman.es.port}
>>>>> apiman-gateway.metrics.client.username=${apiman.es.username}
>>>>> apiman-gateway.metrics.client.password=${apiman.es.password}
>>>>> apiman-gateway.metrics.client.timeout=${apiman.es.timeout}
>>>>> apiman-gateway.metrics.client.initialize=true
>>>>>
>>>>> On Fri, Feb 3, 2017 at 11:40 AM, Marc Savy <marc.savy at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> Which metrics implementation are you using? The data you're talking
>>>>>> about should be in there; if it's not then there's a problem.
>>>>>>
>>>>>>
>>>>>> On 3 February 2017 at 09:50, Balu S <sbalu27 at gmail.com> wrote:
>>>>>>
>>>>>>> Thanks for your inputs.
>>>>>>>
>>>>>>> Yes, I mean HTTP error codes (non-200) that are returned to client.
>>>>>>> For example, when a request is missing query parameters and the API
>>>>>>> responds with a "Bad request" (400) along with error XML. Here the error
>>>>>>> code is set at HTTP level only and Apiman metrics should consider it as bad
>>>>>>> response (in my opinion). Neither I do not see the Apiman source
>>>>>>> (HttpApiConnection) interpreting the HTTP response using the HTTP code. But
>>>>>>> with the custom policy, If I check for non-200 code and handle as failure,
>>>>>>> then metrics shows them as error.
>>>>>>>
>>>>>>> To clarify on my implementation with custom policy, I'm not trying
>>>>>>> to change the HTTP error code based on the response, rather we are
>>>>>>> unpacking the error response body and packing in different XML format as
>>>>>>> done by Apiman. Is this not a valid scenario ? I think there could other
>>>>>>> scenarios where one want to alter the response body. I agree there will be
>>>>>>> additional cost to performance and memory, but can it be not done one
>>>>>>> demand basis like how one can implement IDataPolicy to parse the response
>>>>>>> only if he needs to.
>>>>>>>
>>>>>>> Thanks
>>>>>>> Balu
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Feb 2, 2017 at 10:31 PM, Eric Wittmann <
>>>>>>> eric.wittmann at redhat.com> wrote:
>>>>>>>
>>>>>>>> The bottom line here is that you cannot return a Policy Failure (or
>>>>>>>> customize it) based on information in the response body.  The response is
>>>>>>>> streamed from the back-end to the client, and at the time streaming begins,
>>>>>>>> the response code and HTTP headers have already been sent.
>>>>>>>>
>>>>>>>> It sounds to me like you're asking for a feature where you can
>>>>>>>> parse the response body *before* the policy's "apply" method is invoked.
>>>>>>>> We have such a feature for requests, but not for responses.  I suspect core
>>>>>>>> changes to apiman would be required to enable that.  It seems like a
>>>>>>>> reasonable request to me, as long as users of the feature understand the
>>>>>>>> performance and memory requirements of enabling it.
>>>>>>>>
>>>>>>>> -Eric
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Feb 2, 2017 at 1:03 PM, Marc Savy <marc.savy at redhat.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> NB: This is distinct from the body you're setting which contains a
>>>>>>>>> JSON/XML payload containing the error code. It's in the HTTP protocol
>>>>>>>>> itself.
>>>>>>>>>
>>>>>>>>> On 2 February 2017 at 18:02, Marc Savy <marc.savy at redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> That sounds like metrics are going wrong, or perhaps you're
>>>>>>>>>> misinterpreting it  (adding Eric).
>>>>>>>>>>
>>>>>>>>>> When you say your API returns an error, does it still return an
>>>>>>>>>> appropriate non-200 error code at the HTTP level? For instance, 500 or
>>>>>>>>>> similar? That's very important.
>>>>>>>>>>
>>>>>>>>>> There's a difference between an error and a failure - have you
>>>>>>>>>> checked both of those fields to see whether they contain the information
>>>>>>>>>> you're expecting to see.
>>>>>>>>>>
>>>>>>>>>> Certainly in my experience we *do* collect the metrics you're
>>>>>>>>>> talking about, unless I'm misunderstanding you.
>>>>>>>>>>
>>>>>>>>>> On 2 February 2017 at 17:40, Balu S <sbalu27 at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Marc,
>>>>>>>>>>>
>>>>>>>>>>> I shall explain my use case. From our API call, we return error
>>>>>>>>>>> response (in XML or JSON)  for 401, 500 and so on. However, Apiman metrics
>>>>>>>>>>> seems to just consider them as good response (it is, as Apiman received the
>>>>>>>>>>> response back) and show as successful response. So I have made custom
>>>>>>>>>>> policy to intercept the response to know if it is failure and trigger the
>>>>>>>>>>> Policy Failure. This is fairly simple and straight forward as the response
>>>>>>>>>>> code will just do the purpose. But I want also to add some additional
>>>>>>>>>>> information about the failure to Policy Failure. This additional
>>>>>>>>>>> information is in the original error response which will be lost once
>>>>>>>>>>> doFailure() happens. And no we don't want to those additional information
>>>>>>>>>>> in some HTTP headers to pass around. Hence I implemented responseHandler()
>>>>>>>>>>> to handle the response buffer and like you pointed out, it seems to be too
>>>>>>>>>>> late to meddle the response.
>>>>>>>>>>>
>>>>>>>>>>> So ideally, there are possible 2 solution
>>>>>>>>>>>
>>>>>>>>>>>    - Apiman metrics can interpret the response as unsuccessful
>>>>>>>>>>> for such error response from API call.
>>>>>>>>>>>    - Handle the response buffer data before the write() call to
>>>>>>>>>>> response outputstream.
>>>>>>>>>>>
>>>>>>>>>>> Do you see any alternative solution?
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Balu
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Feb 2, 2017 at 6:15 PM, Marc Savy <marc.savy at redhat.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Perhaps URLRewritingPolicy https://github.com/apiman/apim
>>>>>>>>>>>> an/blob/master/gateway/engine/policies/src/main/java/io/apim
>>>>>>>>>>>> an/gateway/engine/policies/URLRewritingPolicy.java be an
>>>>>>>>>>>> informative place to start?
>>>>>>>>>>>>
>>>>>>>>>>>> - Apiman streams data, so the client may be receiving data
>>>>>>>>>>>> already by the time you've determined you want to cancel (the connection is
>>>>>>>>>>>> already established; headers have been sent) - it's often too late to
>>>>>>>>>>>> gracefully cancel. You could try throwing an exception and seeing what
>>>>>>>>>>>> happens (not recommended practice!).
>>>>>>>>>>>>
>>>>>>>>>>>> If that doesn't work, perhaps you can explain your use-case
>>>>>>>>>>>> more clearly and explicitly so we can see what the alternatives are?
>>>>>>>>>>>>
>>>>>>>>>>>> - Policies are *static* instances, if you are assigning that
>>>>>>>>>>>> buffer to the object then it's as if you were writing "static Buffer
>>>>>>>>>>>>  buffer" and different requests will all share that variable (and thus swap
>>>>>>>>>>>> it out repeatedly!).
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Marc
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 2 February 2017 at 16:56, Balu S <sbalu27 at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>> I'm trying to parse the response using responseDataHandler()
>>>>>>>>>>>>> in the custom policy. In cases, if the response from API is of certain
>>>>>>>>>>>>> content, I would like the Apiman to consider as failure. But I don't find a
>>>>>>>>>>>>> way to throw policy failure from responseDataHandler(). And I cannot
>>>>>>>>>>>>> achieve this in doApply() as the ApiResponse object does not have "content"
>>>>>>>>>>>>> to parse.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also, what I found is write(chunk) in the AbstractStream is
>>>>>>>>>>>>> called after doApply, so I cannot set any attributes in it to fetch it in
>>>>>>>>>>>>> doApply() and trigger doFailure().
>>>>>>>>>>>>>
>>>>>>>>>>>>> For example, in below call, how to throw as policy failure
>>>>>>>>>>>>> after parsing the contents ? Or how can I access response content even
>>>>>>>>>>>>> before write() method.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *URLRewritingPolicy.java*
>>>>>>>>>>>>>
>>>>>>>>>>>>>     @Override
>>>>>>>>>>>>>     protected IReadWriteStream<ApiResponse>
>>>>>>>>>>>>> responseDataHandler(ApiResponse response,
>>>>>>>>>>>>>             IPolicyContext context, URLRewritingConfig
>>>>>>>>>>>>> policyConfiguration) {
>>>>>>>>>>>>>         if (policyConfiguration.isProcessResponseBody()) {
>>>>>>>>>>>>>             return new URLRewritingStream(context.get
>>>>>>>>>>>>> Component(IBufferFactoryComponent.class), response,
>>>>>>>>>>>>>                     policyConfiguration.getFromRegex(),
>>>>>>>>>>>>> policyConfiguration.getToReplacement());
>>>>>>>>>>>>>         } else {
>>>>>>>>>>>>>             return null;
>>>>>>>>>>>>>         }
>>>>>>>>>>>>>     }
>>>>>>>>>>>>>
>>>>>>>>>>>>> *URLRewritingStream.java*
>>>>>>>>>>>>>
>>>>>>>>>>>>>     /**
>>>>>>>>>>>>>      * @see io.apiman.gateway.engine.io.Ab
>>>>>>>>>>>>> stractStream#write(io.apiman.gateway.engine.io.IApimanBuffer)
>>>>>>>>>>>>>      */
>>>>>>>>>>>>>     @Override
>>>>>>>>>>>>>     public void write(IApimanBuffer chunk) {
>>>>>>>>>>>>>         if (buffer == null) {
>>>>>>>>>>>>>             buffer = bufferFactory.cloneBuffer(chunk);
>>>>>>>>>>>>>         } else {
>>>>>>>>>>>>>             buffer.append(chunk);
>>>>>>>>>>>>>         }
>>>>>>>>>>>>>         atEnd = false;
>>>>>>>>>>>>>         processBuffer();
>>>>>>>>>>>>>     }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>> Balu
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Apiman-user mailing list
>>>>>>>>>>>>> Apiman-user at lists.jboss.org
>>>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/apiman-user
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.jboss.org/pipermail/apiman-user/attachments/20170204/9e1d82ad/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 54197 bytes
Desc: not available
Url : http://lists.jboss.org/pipermail/apiman-user/attachments/20170204/9e1d82ad/attachment-0001.png 


More information about the Apiman-user mailing list