<div dir="ltr">Hi Eric,<div>yes that looks promising solution for us. I shall try and come back.Thanks</div><div><br></div><div>regards</div><div>Balu</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 3, 2017 at 10:31 PM, Eric Wittmann <span dir="ltr"><<a href="mailto:eric.wittmann@redhat.com" target="_blank">eric.wittmann@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">So I think at least one aspect of this is solvable. You can modify the request metric data directly in a custom policy. You can access and modify the request metric like this:<div><br></div><div>{code}</div><div><div> protected void doApply(ApiResponse response, IPolicyContext context, C config, IPolicyChain<ApiResponse> chain) {</div><div> <span class="m_-361306066014341103gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>RequestMetric metric = context.getAttribute("apiman.<wbr>request-metric", null);</div><div> <span class="m_-361306066014341103gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>metric.setFailure(true);</div><div> <span class="m_-361306066014341103gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>metric.setFailureCode(12345);</div><div> <span class="m_-361306066014341103gmail-Apple-tab-span" style="white-space:pre-wrap"> </span>metric.setFailureReason("<wbr>Backend API failed for a reason!"";</div><div> chain.doApply(response);<br></div><div> }</div></div><div>{code}</div><div><br></div><div>That will allow you to make sure the metrics information is correct for your use-case. The policy context (and thus the request metric object) is also available to data policies:</div><div><br></div><div><a href="https://github.com/apiman/apiman/blob/master/gateway/engine/core/src/main/java/io/apiman/gateway/engine/policy/IDataPolicy.java#L53-L54" target="_blank">https://github.com/apiman/<wbr>apiman/blob/master/gateway/<wbr>engine/core/src/main/java/io/<wbr>apiman/gateway/engine/policy/<wbr>IDataPolicy.java#L53-L54</a><br></div><div><br></div><div>Does that make sense? Is there more to your use-case?</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>-Eric</div></font></span><div><div class="h5"><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 3, 2017 at 6:48 AM, Marc Savy <span dir="ltr"><<a href="mailto:marc.savy@redhat.com" target="_blank">marc.savy@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Aha, you're specifically talking about the apiman UI - I had thought from the context that you were using an external tool like Grafana. <div><br></div><div>I'll let Eric respond as he implemented that code.</div></div><div class="m_-361306066014341103gmail-HOEnZb"><div class="m_-361306066014341103gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On 3 February 2017 at 11:38, Balu S <span dir="ltr"><<a href="mailto:sbalu27@gmail.com" target="_blank">sbalu27@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">sorry, I don't understand what you wrote.<div><br></div><div>I would like Apiman metrics to consider the HTTP response (>=400) as failure/error (whatever that is appropriate) and indicate in below charts.</div><div><br></div><div><br></div><div><br></div><div><img src="cid:ii_15a03c3d5bc6a8bb" alt="Inline image 1" width="472" height="275"><br></div></div><div class="m_-361306066014341103gmail-m_-7818692092858903955HOEnZb"><div class="m_-361306066014341103gmail-m_-7818692092858903955h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 3, 2017 at 12:21 PM, Marc Savy <span dir="ltr"><<a href="mailto:marc.savy@redhat.com" target="_blank">marc.savy@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I think I understand what you're asking.<div><br></div><div>IIRC:<br><div><br></div><div>Error = an exception occurred within the gateway (e.g. could not connect to backend).</div><div>Failure = a policy failure occurred.</div><div><br></div><div><br></div><div>If you're wanting to look at http exception codes, just look at the 'responseCode' field in the ES Metrics.</div><div><br></div><div>If you're wanting a "failed/not failed" determination then perhaps just execute `responseCode/100 == 2` in your analysis? Or is this not possible in your use-case?</div></div><div><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679h5"><div class="gmail_extra"><br><div class="gmail_quote">On 3 February 2017 at 10:45, Balu S <span dir="ltr"><<a href="mailto:sbalu27@gmail.com" target="_blank">sbalu27@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Its the default one ES.<div><br></div><div><div># ------------------------------<wbr>------------------------------<wbr>---------</div><div># Elasticsearch Metrics Settings</div><div># ------------------------------<wbr>------------------------------<wbr>---------</div><div><br></div><div>apiman-gateway.metrics=io.<a href="http://apiman.gateway.engine.es" target="_blank">apim<wbr>an.gateway.engine.es</a>.ESMetrics</div><div>apiman-gateway.metrics.client.<wbr>type=jest</div><div>apiman-gateway.metrics.client.<wbr>protocol=${apiman.es.protocol}</div><div>apiman-gateway.metrics.client.<wbr>host=${apiman.es.host}</div><div>apiman-gateway.metrics.client.<wbr>port=${apiman.es.port}</div><div>apiman-gateway.metrics.client.<wbr>username=${apiman.es.username}</div><div>apiman-gateway.metrics.client.<wbr>password=${apiman.es.password}</div><div>apiman-gateway.metrics.client.<wbr>timeout=${apiman.es.timeout}</div><div>apiman-gateway.metrics.client.<wbr>initialize=true</div></div></div><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360HOEnZb"><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 3, 2017 at 11:40 AM, Marc Savy <span dir="ltr"><<a href="mailto:marc.savy@redhat.com" target="_blank">marc.savy@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Which metrics implementation are you using? The data you're talking about should be in there; if it's not then there's a problem.</div><div><br></div></div><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225HOEnZb"><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225h5"><div class="gmail_extra"><br><div class="gmail_quote">On 3 February 2017 at 09:50, Balu S <span dir="ltr"><<a href="mailto:sbalu27@gmail.com" target="_blank">sbalu27@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thanks for your inputs.<div><br></div><div>Yes, I mean HTTP error codes (non-200) that are returned to client. For example, when a request is missing query parameters and the API responds with a "Bad request" (400) along with error XML. Here the error code is set at HTTP level only and Apiman metrics should consider it as bad response (in my opinion). Neither I do not see the Apiman source (HttpApiConnection) interpreting the HTTP response using the HTTP code. But with the custom policy, If I check for non-200 code and handle as failure, then metrics shows them as error.</div><div><br></div><div>To clarify on my implementation with custom policy, I'm not trying to change the HTTP error code based on the response, rather we are unpacking the error response body and packing in different XML format as done by Apiman. Is this not a valid scenario ? I think there could other scenarios where one want to alter the response body. I agree there will be additional cost to performance and memory, but can it be not done one demand basis like how one can implement IDataPolicy to parse the response only if he needs to.</div><div><br></div><div>Thanks</div><span class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988HOEnZb"><font color="#888888"><div>Balu</div><div><br></div><div><br></div><div><br></div><div><br></div></font></span></div><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988HOEnZb"><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 2, 2017 at 10:31 PM, Eric Wittmann <span dir="ltr"><<a href="mailto:eric.wittmann@redhat.com" target="_blank">eric.wittmann@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">The bottom line here is that you cannot return a Policy Failure (or customize it) based on information in the response body. The response is streamed from the back-end to the client, and at the time streaming begins, the response code and HTTP headers have already been sent. <div><br></div><div>It sounds to me like you're asking for a feature where you can parse the response body *before* the policy's "apply" method is invoked. We have such a feature for requests, but not for responses. I suspect core changes to apiman would be required to enable that. It seems like a reasonable request to me, as long as users of the feature understand the performance and memory requirements of enabling it. </div><span class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988m_7606916829626292824HOEnZb"><font color="#888888"><div><br></div><div>-Eric</div><div><br></div></font></span></div><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988m_7606916829626292824HOEnZb"><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988m_7606916829626292824h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 2, 2017 at 1:03 PM, Marc Savy <span dir="ltr"><<a href="mailto:marc.savy@redhat.com" target="_blank">marc.savy@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">NB: This is distinct from the body you're setting which contains a JSON/XML payload containing the error code. It's in the HTTP protocol itself.</div><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988m_7606916829626292824m_-8351676212353238146HOEnZb"><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988m_7606916829626292824m_-8351676212353238146h5"><div class="gmail_extra"><br><div class="gmail_quote">On 2 February 2017 at 18:02, Marc Savy <span dir="ltr"><<a href="mailto:marc.savy@redhat.com" target="_blank">marc.savy@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">That sounds like metrics are going wrong, or perhaps you're misinterpreting it (adding Eric). <div><br></div><div>When you say your API returns an error, does it still return an appropriate non-200 error code at the HTTP level? For instance, 500 or similar? That's very important.</div><div><br></div><div>There's a difference between an error and a failure - have you checked both of those fields to see whether they contain the information you're expecting to see.</div><div><br></div><div>Certainly in my experience we *do* collect the metrics you're talking about, unless I'm misunderstanding you. </div></div><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988m_7606916829626292824m_-8351676212353238146m_8225609818623742585HOEnZb"><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988m_7606916829626292824m_-8351676212353238146m_8225609818623742585h5"><div class="gmail_extra"><br><div class="gmail_quote">On 2 February 2017 at 17:40, Balu S <span dir="ltr"><<a href="mailto:sbalu27@gmail.com" target="_blank">sbalu27@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Marc,<div><br></div><div>I shall explain my use case. From our API call, we return error response (in XML or JSON) for 401, 500 and so on. However, Apiman metrics seems to just consider them as good response (it is, as Apiman received the response back) and show as successful response. So I have made custom policy to intercept the response to know if it is failure and trigger the Policy Failure. This is fairly simple and straight forward as the response code will just do the purpose. But I want also to add some additional information about the failure to Policy Failure. This additional information is in the original error response which will be lost once doFailure() happens. And no we don't want to those additional information in some HTTP headers to pass around. Hence I implemented responseHandler() to handle the response buffer and like you pointed out, it seems to be too late to meddle the response. </div><div><br></div><div>So ideally, there are possible 2 solution</div><div><br></div><div> - Apiman metrics can interpret the response as unsuccessful for such error response from API call.</div><div> - Handle the response buffer data before the write() call to response outputstream.</div><div><br></div><div>Do you see any alternative solution? </div><div><br></div><div>Thanks</div><span class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988m_7606916829626292824m_-8351676212353238146m_8225609818623742585m_-2302524732941523296HOEnZb"><font color="#888888"><div>Balu</div></font></span></div><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988m_7606916829626292824m_-8351676212353238146m_8225609818623742585m_-2302524732941523296HOEnZb"><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988m_7606916829626292824m_-8351676212353238146m_8225609818623742585m_-2302524732941523296h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 2, 2017 at 6:15 PM, Marc Savy <span dir="ltr"><<a href="mailto:marc.savy@redhat.com" target="_blank">marc.savy@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>Perhaps URLRewritingPolicy <a href="https://github.com/apiman/apiman/blob/master/gateway/engine/policies/src/main/java/io/apiman/gateway/engine/policies/URLRewritingPolicy.java" target="_blank">https://github.com/apiman/apim<wbr>an/blob/master/gateway/engine/<wbr>policies/src/main/java/io/apim<wbr>an/gateway/engine/policies/URL<wbr>RewritingPolicy.java</a> be an informative place to start?</div><div><br></div><div>- Apiman streams data, so the client may be receiving data already by the time you've determined you want to cancel (the connection is already established; headers have been sent) - it's often too late to gracefully cancel. You could try throwing an exception and seeing what happens (not recommended practice!).</div><div><br></div><div>If that doesn't work, perhaps you can explain your use-case more clearly and explicitly so we can see what the alternatives are?</div><div><br></div><div>- Policies are *static* instances, if you are assigning that buffer to the object then it's as if you were writing "static Buffer buffer" and different requests will all share that variable (and thus swap it out repeatedly!).</div><div><br></div><div>Regards,</div><div>Marc</div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988m_7606916829626292824m_-8351676212353238146m_8225609818623742585m_-2302524732941523296m_5329327770276396523h5">On 2 February 2017 at 16:56, Balu S <span dir="ltr"><<a href="mailto:sbalu27@gmail.com" target="_blank">sbalu27@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988m_7606916829626292824m_-8351676212353238146m_8225609818623742585m_-2302524732941523296m_5329327770276396523h5"><div dir="ltr">Hello,<div>I'm trying to parse the response using responseDataHandler() in the custom policy. In cases, if the response from API is of certain content, I would like the Apiman to consider as failure. But I don't find a way to throw policy failure from responseDataHandler(). And I cannot achieve this in doApply() as the ApiResponse object does not have "content" to parse. </div><div><br></div><div>Also, what I found is write(chunk) in the AbstractStream is called after doApply, so I cannot set any attributes in it to fetch it in doApply() and trigger doFailure().</div><div><br></div><div>For example, in below call, how to throw as policy failure after parsing the contents ? Or how can I access response content even before write() method.</div><div><br></div><div><br></div><div><b>URLRewritingPolicy.java</b><br></div><div><br></div><div><div> @Override</div><div> protected IReadWriteStream<ApiResponse> responseDataHandler(ApiRespons<wbr>e response,</div><div> IPolicyContext context, URLRewritingConfig policyConfiguration) {</div><div> if (policyConfiguration.isProcess<wbr>ResponseBody()) {</div><div> return new URLRewritingStream(context.get<wbr>Component(IBufferFactoryCompon<wbr>ent.class), response,</div><div> policyConfiguration.getFromReg<wbr>ex(), policyConfiguration.getToRepla<wbr>cement());</div><div> } else {</div><div> return null;</div><div> }</div><div> }</div></div><div><b><br></b></div><div><b>URLRewritingStream.java</b><br></div><div><br></div><div><div> /**</div><div> * @see <a href="http://io.apiman.gateway.engine.io" target="_blank">io.apiman.gateway.engine.io</a>.Ab<wbr>stractStream#write(io.<a href="http://apiman.gateway.engine.io" target="_blank">apiman.g<wbr>ateway.engine.io</a>.IApimanBuffer<wbr>)</div><div> */</div><div> @Override</div><div> public void write(IApimanBuffer chunk) {</div><div> if (buffer == null) {</div><div> buffer = bufferFactory.cloneBuffer(chun<wbr>k);</div><div> } else {</div><div> buffer.append(chunk);</div><div> }</div><div> atEnd = false;</div><div> processBuffer();</div><div> }</div></div><div><br></div><div><br></div><div>Best regards</div><span class="m_-361306066014341103gmail-m_-7818692092858903955m_8497495563300498679m_-5297445025373110861m_4204126679526797360m_-9016631226611913225m_6426889429172039988m_7606916829626292824m_-8351676212353238146m_8225609818623742585m_-2302524732941523296m_5329327770276396523m_-3176989166064670582HOEnZb"><font color="#888888"><div>Balu</div></font></span></div>
<br></div></div>______________________________<wbr>_________________<br>
Apiman-user mailing list<br>
<a href="mailto:Apiman-user@lists.jboss.org" target="_blank">Apiman-user@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/apiman-user" rel="noreferrer" target="_blank">https://lists.jboss.org/mailma<wbr>n/listinfo/apiman-user</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>