Thanks for your inputs. Yes, I mean HTTP error codes (non-200) that are returned to client. For example, when a request is missing query parameters and the API responds with a "Bad request" (400) along with error XML. Here the error code is set at HTTP level only and Apiman metrics should consider it as bad response (in my opinion). Neither I do not see the Apiman source (HttpApiConnection) interpreting the HTTP response using the HTTP code. But with the custom policy, If I check for non-200 code and handle as failure, then metrics shows them as error. To clarify on my implementation with custom policy, I'm not trying to change the HTTP error code based on the response, rather we are unpacking the error response body and packing in different XML format as done by Apiman. Is this not a valid scenario ? I think there could other scenarios where one want to alter the response body. I agree there will be additional cost to performance and memory, but can it be not done one demand basis like how one can implement IDataPolicy to parse the response only if he needs to. Thanks Balu On Thu, Feb 2, 2017 at 10:31 PM, Eric Wittmann <eric.wittmann(a)redhat.com&gt; wrote: > The bottom line here is that you cannot return a Policy Failure (or > customize it) based on information in the response body. The response is > streamed from the back-end to the client, and at the time streaming begins, > the response code and HTTP headers have already been sent. > > It sounds to me like you're asking for a feature where you can parse the > response body *before* the policy's "apply" method is invoked. We have > such a feature for requests, but not for responses. I suspect core changes > to apiman would be required to enable that. It seems like a reasonable > request to me, as long as users of the feature understand the performance > and memory requirements of enabling it. > > -Eric > > > On Thu, Feb 2, 2017 at 1:03 PM, Marc Savy <marc.savy(a)redhat.com&gt; wrote: > >> NB: This is distinct from the body you're setting which contains a >> JSON/XML payload containing the error code. It's in the HTTP protocol >> itself. >> >> On 2 February 2017 at 18:02, Marc Savy <marc.savy(a)redhat.com&gt; wrote: >> >>> That sounds like metrics are going wrong, or perhaps you're >>> misinterpreting it (adding Eric). >>> >>> When you say your API returns an error, does it still return an >>> appropriate non-200 error code at the HTTP level? For instance, 500 or >>> similar? That's very important. >>> >>> There's a difference between an error and a failure - have you checked >>> both of those fields to see whether they contain the information you're >>> expecting to see. >>> >>> Certainly in my experience we *do* collect the metrics you're talking >>> about, unless I'm misunderstanding you. >>> >>> On 2 February 2017 at 17:40, Balu S <sbalu27(a)gmail.com&gt; wrote: >>> >>>> Hi Marc, >>>> >>>> I shall explain my use case. From our API call, we return error >>>> response (in XML or JSON) for 401, 500 and so on. However, Apiman metrics >>>> seems to just consider them as good response (it is, as Apiman received the >>>> response back) and show as successful response. So I have made custom >>>> policy to intercept the response to know if it is failure and trigger the >>>> Policy Failure. This is fairly simple and straight forward as the response >>>> code will just do the purpose. But I want also to add some additional >>>> information about the failure to Policy Failure. This additional >>>> information is in the original error response which will be lost once >>>> doFailure() happens. And no we don't want to those additional information >>>> in some HTTP headers to pass around. Hence I implemented responseHandler() >>>> to handle the response buffer and like you pointed out, it seems to be too >>>> late to meddle the response. >>>> >>>> So ideally, there are possible 2 solution >>>> >>>> - Apiman metrics can interpret the response as unsuccessful for >>>> such error response from API call. >>>> - Handle the response buffer data before the write() call to >>>> response outputstream. >>>> >>>> Do you see any alternative solution? >>>> >>>> Thanks >>>> Balu >>>> >>>> On Thu, Feb 2, 2017 at 6:15 PM, Marc Savy <marc.savy(a)redhat.com&gt; >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> Perhaps URLRewritingPolicy https://github.com/apiman/apim >>>>> an/blob/master/gateway/engine/policies/src/main/java/io/apim >>>>> an/gateway/engine/policies/URLRewritingPolicy.java be an informative >>>>> place to start? >>>>> >>>>> - Apiman streams data, so the client may be receiving data already by >>>>> the time you've determined you want to cancel (the connection is already >>>>> established; headers have been sent) - it's often too late to gracefully >>>>> cancel. You could try throwing an exception and seeing what happens (not >>>>> recommended practice!). >>>>> >>>>> If that doesn't work, perhaps you can explain your use-case more >>>>> clearly and explicitly so we can see what the alternatives are? >>>>> >>>>> - Policies are *static* instances, if you are assigning that buffer >>>>> to the object then it's as if you were writing "static Buffer buffer" and >>>>> different requests will all share that variable (and thus swap it out >>>>> repeatedly!). >>>>> >>>>> Regards, >>>>> Marc >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 2 February 2017 at 16:56, Balu S <sbalu27(a)gmail.com&gt; wrote: >>>>> >>>>>> Hello, >>>>>> I'm trying to parse the response using responseDataHandler() in the >>>>>> custom policy. In cases, if the response from API is of certain content, I >>>>>> would like the Apiman to consider as failure. But I don't find a way to >>>>>> throw policy failure from responseDataHandler(). And I cannot achieve this >>>>>> in doApply() as the ApiResponse object does not have "content" to parse. >>>>>> >>>>>> Also, what I found is write(chunk) in the AbstractStream is called >>>>>> after doApply, so I cannot set any attributes in it to fetch it in >>>>>> doApply() and trigger doFailure(). >>>>>> >>>>>> For example, in below call, how to throw as policy failure after >>>>>> parsing the contents ? Or how can I access response content even before >>>>>> write() method. >>>>>> >>>>>> >>>>>> *URLRewritingPolicy.java* >>>>>> >>>>>> @Override >>>>>> protected IReadWriteStream<ApiResponse> >>>>>> responseDataHandler(ApiResponse response, >>>>>> IPolicyContext context, URLRewritingConfig >>>>>> policyConfiguration) { >>>>>> if (policyConfiguration.isProcessResponseBody()) { >>>>>> return new URLRewritingStream(context.get >>>>>> Component(IBufferFactoryComponent.class), response, >>>>>> policyConfiguration.getFromRegex(), >>>>>> policyConfiguration.getToReplacement()); >>>>>> } else { >>>>>> return null; >>>>>> } >>>>>> } >>>>>> >>>>>> *URLRewritingStream.java* >>>>>> >>>>>> /** >>>>>> * @see io.apiman.gateway.engine.io.AbstractStream#write(io. >>>>>> apiman.gateway.engine.io.IApimanBuffer) >>>>>> */ >>>>>> @Override >>>>>> public void write(IApimanBuffer chunk) { >>>>>> if (buffer == null) { >>>>>> buffer = bufferFactory.cloneBuffer(chunk); >>>>>> } else { >>>>>> buffer.append(chunk); >>>>>> } >>>>>> atEnd = false; >>>>>> processBuffer(); >>>>>> } >>>>>> >>>>>> >>>>>> Best regards >>>>>> Balu >>>>>> >>>>>> _______________________________________________ >>>>>> Apiman-user mailing list >>>>>> Apiman-user(a)lists.jboss.org >>>>>> https://lists.jboss.org/mailman/listinfo/apiman-user >>>>>> >>>>>> >>>>> >>>> >>> >> >

Its the default one ES. # --------------------------------------------------------------------- # Elasticsearch Metrics Settings # --------------------------------------------------------------------- apiman-gateway.metrics=io.apiman.gateway.engine.es.ESMetrics apiman-gateway.metrics.client.type=jest apiman-gateway.metrics.client.protocol=${apiman.es.protocol} apiman-gateway.metrics.client.host=${apiman.es.host} apiman-gateway.metrics.client.port=${apiman.es.port} apiman-gateway.metrics.client.username=${apiman.es.username} apiman-gateway.metrics.client.password=${apiman.es.password} apiman-gateway.metrics.client.timeout=${apiman.es.timeout} apiman-gateway.metrics.client.initialize=true On Fri, Feb 3, 2017 at 11:40 AM, Marc Savy <marc.savy(a)redhat.com&gt; wrote: > Which metrics implementation are you using? The data you're talking about > should be in there; if it's not then there's a problem. > > > On 3 February 2017 at 09:50, Balu S <sbalu27(a)gmail.com&gt; wrote: > >> Thanks for your inputs. >> >> Yes, I mean HTTP error codes (non-200) that are returned to client. For >> example, when a request is missing query parameters and the API responds >> with a "Bad request" (400) along with error XML. Here the error code is set >> at HTTP level only and Apiman metrics should consider it as bad response >> (in my opinion). Neither I do not see the Apiman source (HttpApiConnection) >> interpreting the HTTP response using the HTTP code. But with the custom >> policy, If I check for non-200 code and handle as failure, then metrics >> shows them as error. >> >> To clarify on my implementation with custom policy, I'm not trying to >> change the HTTP error code based on the response, rather we are unpacking >> the error response body and packing in different XML format as done by >> Apiman. Is this not a valid scenario ? I think there could other scenarios >> where one want to alter the response body. I agree there will be additional >> cost to performance and memory, but can it be not done one demand basis >> like how one can implement IDataPolicy to parse the response only if he >> needs to. >> >> Thanks >> Balu >> >> >> >> >> >> On Thu, Feb 2, 2017 at 10:31 PM, Eric Wittmann <eric.wittmann(a)redhat.com >> > wrote: >> >>> The bottom line here is that you cannot return a Policy Failure (or >>> customize it) based on information in the response body. The response is >>> streamed from the back-end to the client, and at the time streaming begins, >>> the response code and HTTP headers have already been sent. >>> >>> It sounds to me like you're asking for a feature where you can parse >>> the response body *before* the policy's "apply" method is invoked. We have >>> such a feature for requests, but not for responses. I suspect core changes >>> to apiman would be required to enable that. It seems like a reasonable >>> request to me, as long as users of the feature understand the performance >>> and memory requirements of enabling it. >>> >>> -Eric >>> >>> >>> On Thu, Feb 2, 2017 at 1:03 PM, Marc Savy <marc.savy(a)redhat.com&gt; wrote: >>> >>>> NB: This is distinct from the body you're setting which contains a >>>> JSON/XML payload containing the error code. It's in the HTTP protocol >>>> itself. >>>> >>>> On 2 February 2017 at 18:02, Marc Savy <marc.savy(a)redhat.com&gt; wrote: >>>> >>>>> That sounds like metrics are going wrong, or perhaps you're >>>>> misinterpreting it (adding Eric). >>>>> >>>>> When you say your API returns an error, does it still return an >>>>> appropriate non-200 error code at the HTTP level? For instance, 500 or >>>>> similar? That's very important. >>>>> >>>>> There's a difference between an error and a failure - have you >>>>> checked both of those fields to see whether they contain the information >>>>> you're expecting to see. >>>>> >>>>> Certainly in my experience we *do* collect the metrics you're talking >>>>> about, unless I'm misunderstanding you. >>>>> >>>>> On 2 February 2017 at 17:40, Balu S <sbalu27(a)gmail.com&gt; wrote: >>>>> >>>>>> Hi Marc, >>>>>> >>>>>> I shall explain my use case. From our API call, we return error >>>>>> response (in XML or JSON) for 401, 500 and so on. However, Apiman metrics >>>>>> seems to just consider them as good response (it is, as Apiman received the >>>>>> response back) and show as successful response. So I have made custom >>>>>> policy to intercept the response to know if it is failure and trigger the >>>>>> Policy Failure. This is fairly simple and straight forward as the response >>>>>> code will just do the purpose. But I want also to add some additional >>>>>> information about the failure to Policy Failure. This additional >>>>>> information is in the original error response which will be lost once >>>>>> doFailure() happens. And no we don't want to those additional information >>>>>> in some HTTP headers to pass around. Hence I implemented responseHandler() >>>>>> to handle the response buffer and like you pointed out, it seems to be too >>>>>> late to meddle the response. >>>>>> >>>>>> So ideally, there are possible 2 solution >>>>>> >>>>>> - Apiman metrics can interpret the response as unsuccessful for >>>>>> such error response from API call. >>>>>> - Handle the response buffer data before the write() call to >>>>>> response outputstream. >>>>>> >>>>>> Do you see any alternative solution? >>>>>> >>>>>> Thanks >>>>>> Balu >>>>>> >>>>>> On Thu, Feb 2, 2017 at 6:15 PM, Marc Savy <marc.savy(a)redhat.com&gt; >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Perhaps URLRewritingPolicy https://github.com/apiman/apim >>>>>>> an/blob/master/gateway/engine/policies/src/main/java/io/apim >>>>>>> an/gateway/engine/policies/URLRewritingPolicy.java be an >>>>>>> informative place to start? >>>>>>> >>>>>>> - Apiman streams data, so the client may be receiving data already >>>>>>> by the time you've determined you want to cancel (the connection is already >>>>>>> established; headers have been sent) - it's often too late to gracefully >>>>>>> cancel. You could try throwing an exception and seeing what happens (not >>>>>>> recommended practice!). >>>>>>> >>>>>>> If that doesn't work, perhaps you can explain your use-case more >>>>>>> clearly and explicitly so we can see what the alternatives are? >>>>>>> >>>>>>> - Policies are *static* instances, if you are assigning that buffer >>>>>>> to the object then it's as if you were writing "static Buffer buffer" and >>>>>>> different requests will all share that variable (and thus swap it out >>>>>>> repeatedly!). >>>>>>> >>>>>>> Regards, >>>>>>> Marc >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 2 February 2017 at 16:56, Balu S <sbalu27(a)gmail.com&gt; wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> I'm trying to parse the response using responseDataHandler() in >>>>>>>> the custom policy. In cases, if the response from API is of certain >>>>>>>> content, I would like the Apiman to consider as failure. But I don't find a >>>>>>>> way to throw policy failure from responseDataHandler(). And I cannot >>>>>>>> achieve this in doApply() as the ApiResponse object does not have "content" >>>>>>>> to parse. >>>>>>>> >>>>>>>> Also, what I found is write(chunk) in the AbstractStream is called >>>>>>>> after doApply, so I cannot set any attributes in it to fetch it in >>>>>>>> doApply() and trigger doFailure(). >>>>>>>> >>>>>>>> For example, in below call, how to throw as policy failure after >>>>>>>> parsing the contents ? Or how can I access response content even before >>>>>>>> write() method. >>>>>>>> >>>>>>>> >>>>>>>> *URLRewritingPolicy.java* >>>>>>>> >>>>>>>> @Override >>>>>>>> protected IReadWriteStream<ApiResponse> >>>>>>>> responseDataHandler(ApiResponse response, >>>>>>>> IPolicyContext context, URLRewritingConfig >>>>>>>> policyConfiguration) { >>>>>>>> if (policyConfiguration.isProcessResponseBody()) { >>>>>>>> return new URLRewritingStream(context.get >>>>>>>> Component(IBufferFactoryComponent.class), response, >>>>>>>> policyConfiguration.getFromRegex(), >>>>>>>> policyConfiguration.getToReplacement()); >>>>>>>> } else { >>>>>>>> return null; >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> *URLRewritingStream.java* >>>>>>>> >>>>>>>> /** >>>>>>>> * @see io.apiman.gateway.engine.io.AbstractStream#write(io. >>>>>>>> apiman.gateway.engine.io.IApimanBuffer) >>>>>>>> */ >>>>>>>> @Override >>>>>>>> public void write(IApimanBuffer chunk) { >>>>>>>> if (buffer == null) { >>>>>>>> buffer = bufferFactory.cloneBuffer(chunk); >>>>>>>> } else { >>>>>>>> buffer.append(chunk); >>>>>>>> } >>>>>>>> atEnd = false; >>>>>>>> processBuffer(); >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> Best regards >>>>>>>> Balu >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Apiman-user mailing list >>>>>>>> Apiman-user(a)lists.jboss.org >>>>>>>> https://lists.jboss.org/mailman/listinfo/apiman-user >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >

sorry, I don't understand what you wrote. I would like Apiman metrics to consider the HTTP response (>=400) as failure/error (whatever that is appropriate) and indicate in below charts. [image: Inline image 1] On Fri, Feb 3, 2017 at 12:21 PM, Marc Savy <marc.savy(a)redhat.com&gt; wrote: > I think I understand what you're asking. > > IIRC: > > Error = an exception occurred within the gateway (e.g. could not connect > to backend). > Failure = a policy failure occurred. > > > If you're wanting to look at http exception codes, just look at the > 'responseCode' field in the ES Metrics. > > If you're wanting a "failed/not failed" determination then perhaps just > execute `responseCode/100 == 2` in your analysis? Or is this not possible > in your use-case? > > On 3 February 2017 at 10:45, Balu S <sbalu27(a)gmail.com&gt; wrote: > >> Its the default one ES. >> >> # --------------------------------------------------------------------- >> # Elasticsearch Metrics Settings >> # --------------------------------------------------------------------- >> >> apiman-gateway.metrics=io.apiman.gateway.engine.es.ESMetrics >> apiman-gateway.metrics.client.type=jest >> apiman-gateway.metrics.client.protocol=${apiman.es.protocol} >> apiman-gateway.metrics.client.host=${apiman.es.host} >> apiman-gateway.metrics.client.port=${apiman.es.port} >> apiman-gateway.metrics.client.username=${apiman.es.username} >> apiman-gateway.metrics.client.password=${apiman.es.password} >> apiman-gateway.metrics.client.timeout=${apiman.es.timeout} >> apiman-gateway.metrics.client.initialize=true >> >> On Fri, Feb 3, 2017 at 11:40 AM, Marc Savy <marc.savy(a)redhat.com&gt; wrote: >> >>> Which metrics implementation are you using? The data you're talking >>> about should be in there; if it's not then there's a problem. >>> >>> >>> On 3 February 2017 at 09:50, Balu S <sbalu27(a)gmail.com&gt; wrote: >>> >>>> Thanks for your inputs. >>>> >>>> Yes, I mean HTTP error codes (non-200) that are returned to client. >>>> For example, when a request is missing query parameters and the API >>>> responds with a "Bad request" (400) along with error XML. Here the error >>>> code is set at HTTP level only and Apiman metrics should consider it as bad >>>> response (in my opinion). Neither I do not see the Apiman source >>>> (HttpApiConnection) interpreting the HTTP response using the HTTP code. But >>>> with the custom policy, If I check for non-200 code and handle as failure, >>>> then metrics shows them as error. >>>> >>>> To clarify on my implementation with custom policy, I'm not trying to >>>> change the HTTP error code based on the response, rather we are unpacking >>>> the error response body and packing in different XML format as done by >>>> Apiman. Is this not a valid scenario ? I think there could other scenarios >>>> where one want to alter the response body. I agree there will be additional >>>> cost to performance and memory, but can it be not done one demand basis >>>> like how one can implement IDataPolicy to parse the response only if he >>>> needs to. >>>> >>>> Thanks >>>> Balu >>>> >>>> >>>> >>>> >>>> >>>> On Thu, Feb 2, 2017 at 10:31 PM, Eric Wittmann < >>>> eric.wittmann(a)redhat.com&gt; wrote: >>>> >>>>> The bottom line here is that you cannot return a Policy Failure (or >>>>> customize it) based on information in the response body. The response is >>>>> streamed from the back-end to the client, and at the time streaming begins, >>>>> the response code and HTTP headers have already been sent. >>>>> >>>>> It sounds to me like you're asking for a feature where you can parse >>>>> the response body *before* the policy's "apply" method is invoked. We have >>>>> such a feature for requests, but not for responses. I suspect core changes >>>>> to apiman would be required to enable that. It seems like a reasonable >>>>> request to me, as long as users of the feature understand the performance >>>>> and memory requirements of enabling it. >>>>> >>>>> -Eric >>>>> >>>>> >>>>> On Thu, Feb 2, 2017 at 1:03 PM, Marc Savy <marc.savy(a)redhat.com&gt; >>>>> wrote: >>>>> >>>>>> NB: This is distinct from the body you're setting which contains a >>>>>> JSON/XML payload containing the error code. It's in the HTTP protocol >>>>>> itself. >>>>>> >>>>>> On 2 February 2017 at 18:02, Marc Savy <marc.savy(a)redhat.com&gt; wrote: >>>>>> >>>>>>> That sounds like metrics are going wrong, or perhaps you're >>>>>>> misinterpreting it (adding Eric). >>>>>>> >>>>>>> When you say your API returns an error, does it still return an >>>>>>> appropriate non-200 error code at the HTTP level? For instance, 500 or >>>>>>> similar? That's very important. >>>>>>> >>>>>>> There's a difference between an error and a failure - have you >>>>>>> checked both of those fields to see whether they contain the information >>>>>>> you're expecting to see. >>>>>>> >>>>>>> Certainly in my experience we *do* collect the metrics you're >>>>>>> talking about, unless I'm misunderstanding you. >>>>>>> >>>>>>> On 2 February 2017 at 17:40, Balu S <sbalu27(a)gmail.com&gt; wrote: >>>>>>> >>>>>>>> Hi Marc, >>>>>>>> >>>>>>>> I shall explain my use case. From our API call, we return error >>>>>>>> response (in XML or JSON) for 401, 500 and so on. However, Apiman metrics >>>>>>>> seems to just consider them as good response (it is, as Apiman received the >>>>>>>> response back) and show as successful response. So I have made custom >>>>>>>> policy to intercept the response to know if it is failure and trigger the >>>>>>>> Policy Failure. This is fairly simple and straight forward as the response >>>>>>>> code will just do the purpose. But I want also to add some additional >>>>>>>> information about the failure to Policy Failure. This additional >>>>>>>> information is in the original error response which will be lost once >>>>>>>> doFailure() happens. And no we don't want to those additional information >>>>>>>> in some HTTP headers to pass around. Hence I implemented responseHandler() >>>>>>>> to handle the response buffer and like you pointed out, it seems to be too >>>>>>>> late to meddle the response. >>>>>>>> >>>>>>>> So ideally, there are possible 2 solution >>>>>>>> >>>>>>>> - Apiman metrics can interpret the response as unsuccessful for >>>>>>>> such error response from API call. >>>>>>>> - Handle the response buffer data before the write() call to >>>>>>>> response outputstream. >>>>>>>> >>>>>>>> Do you see any alternative solution? >>>>>>>> >>>>>>>> Thanks >>>>>>>> Balu >>>>>>>> >>>>>>>> On Thu, Feb 2, 2017 at 6:15 PM, Marc Savy <marc.savy(a)redhat.com&gt; >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Perhaps URLRewritingPolicy https://github.com/apiman/apim >>>>>>>>> an/blob/master/gateway/engine/policies/src/main/java/io/apim >>>>>>>>> an/gateway/engine/policies/URLRewritingPolicy.java be an >>>>>>>>> informative place to start? >>>>>>>>> >>>>>>>>> - Apiman streams data, so the client may be receiving data >>>>>>>>> already by the time you've determined you want to cancel (the connection is >>>>>>>>> already established; headers have been sent) - it's often too late to >>>>>>>>> gracefully cancel. You could try throwing an exception and seeing what >>>>>>>>> happens (not recommended practice!). >>>>>>>>> >>>>>>>>> If that doesn't work, perhaps you can explain your use-case more >>>>>>>>> clearly and explicitly so we can see what the alternatives are? >>>>>>>>> >>>>>>>>> - Policies are *static* instances, if you are assigning that >>>>>>>>> buffer to the object then it's as if you were writing "static Buffer >>>>>>>>> buffer" and different requests will all share that variable (and thus swap >>>>>>>>> it out repeatedly!). >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Marc >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 2 February 2017 at 16:56, Balu S <sbalu27(a)gmail.com&gt; wrote: >>>>>>>>> >>>>>>>>>> Hello, >>>>>>>>>> I'm trying to parse the response using responseDataHandler() in >>>>>>>>>> the custom policy. In cases, if the response from API is of certain >>>>>>>>>> content, I would like the Apiman to consider as failure. But I don't find a >>>>>>>>>> way to throw policy failure from responseDataHandler(). And I cannot >>>>>>>>>> achieve this in doApply() as the ApiResponse object does not have "content" >>>>>>>>>> to parse. >>>>>>>>>> >>>>>>>>>> Also, what I found is write(chunk) in the AbstractStream is >>>>>>>>>> called after doApply, so I cannot set any attributes in it to fetch it in >>>>>>>>>> doApply() and trigger doFailure(). >>>>>>>>>> >>>>>>>>>> For example, in below call, how to throw as policy failure after >>>>>>>>>> parsing the contents ? Or how can I access response content even before >>>>>>>>>> write() method. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> *URLRewritingPolicy.java* >>>>>>>>>> >>>>>>>>>> @Override >>>>>>>>>> protected IReadWriteStream<ApiResponse> >>>>>>>>>> responseDataHandler(ApiResponse response, >>>>>>>>>> IPolicyContext context, URLRewritingConfig >>>>>>>>>> policyConfiguration) { >>>>>>>>>> if (policyConfiguration.isProcessResponseBody()) { >>>>>>>>>> return new URLRewritingStream(context.get >>>>>>>>>> Component(IBufferFactoryComponent.class), response, >>>>>>>>>> policyConfiguration.getFromRegex(), >>>>>>>>>> policyConfiguration.getToReplacement()); >>>>>>>>>> } else { >>>>>>>>>> return null; >>>>>>>>>> } >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> *URLRewritingStream.java* >>>>>>>>>> >>>>>>>>>> /** >>>>>>>>>> * @see io.apiman.gateway.engine.io.AbstractStream#write(io. >>>>>>>>>> apiman.gateway.engine.io.IApimanBuffer) >>>>>>>>>> */ >>>>>>>>>> @Override >>>>>>>>>> public void write(IApimanBuffer chunk) { >>>>>>>>>> if (buffer == null) { >>>>>>>>>> buffer = bufferFactory.cloneBuffer(chunk); >>>>>>>>>> } else { >>>>>>>>>> buffer.append(chunk); >>>>>>>>>> } >>>>>>>>>> atEnd = false; >>>>>>>>>> processBuffer(); >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Best regards >>>>>>>>>> Balu >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Apiman-user mailing list >>>>>>>>>> Apiman-user(a)lists.jboss.org >>>>>>>>>> https://lists.jboss.org/mailman/listinfo/apiman-user >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >

So I think at least one aspect of this is solvable. You can modify the request metric data directly in a custom policy. You can access and modify the request metric like this: {code} protected void doApply(ApiResponse response, IPolicyContext context, C config, IPolicyChain<ApiResponse> chain) { RequestMetric metric = context.getAttribute("apiman.request-metric", null); metric.setFailure(true); metric.setFailureCode(12345); metric.setFailureReason("Backend API failed for a reason!""; chain.doApply(response); } {code} That will allow you to make sure the metrics information is correct for your use-case. The policy context (and thus the request metric object) is also available to data policies: https://github.com/apiman/apiman/blob/master/gateway/ engine/core/src/main/java/io/apiman/gateway/engine/policy/ IDataPolicy.java#L53-L54 Does that make sense? Is there more to your use-case? -Eric On Fri, Feb 3, 2017 at 6:48 AM, Marc Savy <marc.savy(a)redhat.com&gt; wrote: > Aha, you're specifically talking about the apiman UI - I had thought from > the context that you were using an external tool like Grafana. > > I'll let Eric respond as he implemented that code. > > On 3 February 2017 at 11:38, Balu S <sbalu27(a)gmail.com&gt; wrote: > >> sorry, I don't understand what you wrote. >> >> I would like Apiman metrics to consider the HTTP response (>=400) as >> failure/error (whatever that is appropriate) and indicate in below charts. >> >> >> >> [image: Inline image 1] >> >> On Fri, Feb 3, 2017 at 12:21 PM, Marc Savy <marc.savy(a)redhat.com&gt; wrote: >> >>> I think I understand what you're asking. >>> >>> IIRC: >>> >>> Error = an exception occurred within the gateway (e.g. could not >>> connect to backend). >>> Failure = a policy failure occurred. >>> >>> >>> If you're wanting to look at http exception codes, just look at the >>> 'responseCode' field in the ES Metrics. >>> >>> If you're wanting a "failed/not failed" determination then perhaps just >>> execute `responseCode/100 == 2` in your analysis? Or is this not possible >>> in your use-case? >>> >>> On 3 February 2017 at 10:45, Balu S <sbalu27(a)gmail.com&gt; wrote: >>> >>>> Its the default one ES. >>>> >>>> # ------------------------------------------------------------ >>>> --------- >>>> # Elasticsearch Metrics Settings >>>> # ------------------------------------------------------------ >>>> --------- >>>> >>>> apiman-gateway.metrics=io.apiman.gateway.engine.es.ESMetrics >>>> apiman-gateway.metrics.client.type=jest >>>> apiman-gateway.metrics.client.protocol=${apiman.es.protocol} >>>> apiman-gateway.metrics.client.host=${apiman.es.host} >>>> apiman-gateway.metrics.client.port=${apiman.es.port} >>>> apiman-gateway.metrics.client.username=${apiman.es.username} >>>> apiman-gateway.metrics.client.password=${apiman.es.password} >>>> apiman-gateway.metrics.client.timeout=${apiman.es.timeout} >>>> apiman-gateway.metrics.client.initialize=true >>>> >>>> On Fri, Feb 3, 2017 at 11:40 AM, Marc Savy <marc.savy(a)redhat.com&gt; >>>> wrote: >>>> >>>>> Which metrics implementation are you using? The data you're talking >>>>> about should be in there; if it's not then there's a problem. >>>>> >>>>> >>>>> On 3 February 2017 at 09:50, Balu S <sbalu27(a)gmail.com&gt; wrote: >>>>> >>>>>> Thanks for your inputs. >>>>>> >>>>>> Yes, I mean HTTP error codes (non-200) that are returned to client. >>>>>> For example, when a request is missing query parameters and the API >>>>>> responds with a "Bad request" (400) along with error XML. Here the error >>>>>> code is set at HTTP level only and Apiman metrics should consider it as bad >>>>>> response (in my opinion). Neither I do not see the Apiman source >>>>>> (HttpApiConnection) interpreting the HTTP response using the HTTP code. But >>>>>> with the custom policy, If I check for non-200 code and handle as failure, >>>>>> then metrics shows them as error. >>>>>> >>>>>> To clarify on my implementation with custom policy, I'm not trying >>>>>> to change the HTTP error code based on the response, rather we are >>>>>> unpacking the error response body and packing in different XML format as >>>>>> done by Apiman. Is this not a valid scenario ? I think there could other >>>>>> scenarios where one want to alter the response body. I agree there will be >>>>>> additional cost to performance and memory, but can it be not done one >>>>>> demand basis like how one can implement IDataPolicy to parse the response >>>>>> only if he needs to. >>>>>> >>>>>> Thanks >>>>>> Balu >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Feb 2, 2017 at 10:31 PM, Eric Wittmann < >>>>>> eric.wittmann(a)redhat.com&gt; wrote: >>>>>> >>>>>>> The bottom line here is that you cannot return a Policy Failure (or >>>>>>> customize it) based on information in the response body. The response is >>>>>>> streamed from the back-end to the client, and at the time streaming begins, >>>>>>> the response code and HTTP headers have already been sent. >>>>>>> >>>>>>> It sounds to me like you're asking for a feature where you can >>>>>>> parse the response body *before* the policy's "apply" method is invoked. >>>>>>> We have such a feature for requests, but not for responses. I suspect core >>>>>>> changes to apiman would be required to enable that. It seems like a >>>>>>> reasonable request to me, as long as users of the feature understand the >>>>>>> performance and memory requirements of enabling it. >>>>>>> >>>>>>> -Eric >>>>>>> >>>>>>> >>>>>>> On Thu, Feb 2, 2017 at 1:03 PM, Marc Savy <marc.savy(a)redhat.com&gt; >>>>>>> wrote: >>>>>>> >>>>>>>> NB: This is distinct from the body you're setting which contains a >>>>>>>> JSON/XML payload containing the error code. It's in the HTTP protocol >>>>>>>> itself. >>>>>>>> >>>>>>>> On 2 February 2017 at 18:02, Marc Savy <marc.savy(a)redhat.com&gt; >>>>>>>> wrote: >>>>>>>> >>>>>>>>> That sounds like metrics are going wrong, or perhaps you're >>>>>>>>> misinterpreting it (adding Eric). >>>>>>>>> >>>>>>>>> When you say your API returns an error, does it still return an >>>>>>>>> appropriate non-200 error code at the HTTP level? For instance, 500 or >>>>>>>>> similar? That's very important. >>>>>>>>> >>>>>>>>> There's a difference between an error and a failure - have you >>>>>>>>> checked both of those fields to see whether they contain the information >>>>>>>>> you're expecting to see. >>>>>>>>> >>>>>>>>> Certainly in my experience we *do* collect the metrics you're >>>>>>>>> talking about, unless I'm misunderstanding you. >>>>>>>>> >>>>>>>>> On 2 February 2017 at 17:40, Balu S <sbalu27(a)gmail.com&gt; wrote: >>>>>>>>> >>>>>>>>>> Hi Marc, >>>>>>>>>> >>>>>>>>>> I shall explain my use case. From our API call, we return error >>>>>>>>>> response (in XML or JSON) for 401, 500 and so on. However, Apiman metrics >>>>>>>>>> seems to just consider them as good response (it is, as Apiman received the >>>>>>>>>> response back) and show as successful response. So I have made custom >>>>>>>>>> policy to intercept the response to know if it is failure and trigger the >>>>>>>>>> Policy Failure. This is fairly simple and straight forward as the response >>>>>>>>>> code will just do the purpose. But I want also to add some additional >>>>>>>>>> information about the failure to Policy Failure. This additional >>>>>>>>>> information is in the original error response which will be lost once >>>>>>>>>> doFailure() happens. And no we don't want to those additional information >>>>>>>>>> in some HTTP headers to pass around. Hence I implemented responseHandler() >>>>>>>>>> to handle the response buffer and like you pointed out, it seems to be too >>>>>>>>>> late to meddle the response. >>>>>>>>>> >>>>>>>>>> So ideally, there are possible 2 solution >>>>>>>>>> >>>>>>>>>> - Apiman metrics can interpret the response as unsuccessful >>>>>>>>>> for such error response from API call. >>>>>>>>>> - Handle the response buffer data before the write() call to >>>>>>>>>> response outputstream. >>>>>>>>>> >>>>>>>>>> Do you see any alternative solution? >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> Balu >>>>>>>>>> >>>>>>>>>> On Thu, Feb 2, 2017 at 6:15 PM, Marc Savy <marc.savy(a)redhat.com&gt; >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Perhaps URLRewritingPolicy https://github.com/apiman/apim >>>>>>>>>>> an/blob/master/gateway/engine/policies/src/main/java/io/apim >>>>>>>>>>> an/gateway/engine/policies/URLRewritingPolicy.java be an >>>>>>>>>>> informative place to start? >>>>>>>>>>> >>>>>>>>>>> - Apiman streams data, so the client may be receiving data >>>>>>>>>>> already by the time you've determined you want to cancel (the connection is >>>>>>>>>>> already established; headers have been sent) - it's often too late to >>>>>>>>>>> gracefully cancel. You could try throwing an exception and seeing what >>>>>>>>>>> happens (not recommended practice!). >>>>>>>>>>> >>>>>>>>>>> If that doesn't work, perhaps you can explain your use-case >>>>>>>>>>> more clearly and explicitly so we can see what the alternatives are? >>>>>>>>>>> >>>>>>>>>>> - Policies are *static* instances, if you are assigning that >>>>>>>>>>> buffer to the object then it's as if you were writing "static Buffer >>>>>>>>>>> buffer" and different requests will all share that variable (and thus swap >>>>>>>>>>> it out repeatedly!). >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Marc >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 2 February 2017 at 16:56, Balu S <sbalu27(a)gmail.com&gt; wrote: >>>>>>>>>>> >>>>>>>>>>>> Hello, >>>>>>>>>>>> I'm trying to parse the response using responseDataHandler() >>>>>>>>>>>> in the custom policy. In cases, if the response from API is of certain >>>>>>>>>>>> content, I would like the Apiman to consider as failure. But I don't find a >>>>>>>>>>>> way to throw policy failure from responseDataHandler(). And I cannot >>>>>>>>>>>> achieve this in doApply() as the ApiResponse object does not have "content" >>>>>>>>>>>> to parse. >>>>>>>>>>>> >>>>>>>>>>>> Also, what I found is write(chunk) in the AbstractStream is >>>>>>>>>>>> called after doApply, so I cannot set any attributes in it to fetch it in >>>>>>>>>>>> doApply() and trigger doFailure(). >>>>>>>>>>>> >>>>>>>>>>>> For example, in below call, how to throw as policy failure >>>>>>>>>>>> after parsing the contents ? Or how can I access response content even >>>>>>>>>>>> before write() method. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> *URLRewritingPolicy.java* >>>>>>>>>>>> >>>>>>>>>>>> @Override >>>>>>>>>>>> protected IReadWriteStream<ApiResponse> >>>>>>>>>>>> responseDataHandler(ApiResponse response, >>>>>>>>>>>> IPolicyContext context, URLRewritingConfig >>>>>>>>>>>> policyConfiguration) { >>>>>>>>>>>> if (policyConfiguration.isProcessResponseBody()) { >>>>>>>>>>>> return new URLRewritingStream(context.get >>>>>>>>>>>> Component(IBufferFactoryComponent.class), response, >>>>>>>>>>>> policyConfiguration.getFromRegex(), >>>>>>>>>>>> policyConfiguration.getToReplacement()); >>>>>>>>>>>> } else { >>>>>>>>>>>> return null; >>>>>>>>>>>> } >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> *URLRewritingStream.java* >>>>>>>>>>>> >>>>>>>>>>>> /** >>>>>>>>>>>> * @see io.apiman.gateway.engine.io.Ab >>>>>>>>>>>> stractStream#write(io.apiman.gateway.engine.io.IApimanBuffer) >>>>>>>>>>>> */ >>>>>>>>>>>> @Override >>>>>>>>>>>> public void write(IApimanBuffer chunk) { >>>>>>>>>>>> if (buffer == null) { >>>>>>>>>>>> buffer = bufferFactory.cloneBuffer(chunk); >>>>>>>>>>>> } else { >>>>>>>>>>>> buffer.append(chunk); >>>>>>>>>>>> } >>>>>>>>>>>> atEnd = false; >>>>>>>>>>>> processBuffer(); >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Best regards >>>>>>>>>>>> Balu >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Apiman-user mailing list >>>>>>>>>>>> Apiman-user(a)lists.jboss.org >>>>>>>>>>>> https://lists.jboss.org/mailman/listinfo/apiman-user >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >