<p dir="ltr">It's hard to believe the slowdown happens because of Byteman, that should be verified first. It might be because of how metrics are being collected (just guessing as I didn't look) </p>
<p dir="ltr">Great initiative! This looks super useful. <br>
</p>
<div class="gmail_quote">On 3 May 2013 09:36, "Radim Vansa" <<a href="mailto:rvansa@redhat.com">rvansa@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
----- Original Message -----<br>
| From: "Galder Zamarreño" <<a href="mailto:galder@redhat.com">galder@redhat.com</a>><br>
| To: "Radim Vansa" <<a href="mailto:rvansa@redhat.com">rvansa@redhat.com</a>><br>
| Cc: "infinispan-internal Internal" <<a href="mailto:infinispan-internal@redhat.com">infinispan-internal@redhat.com</a>><br>
| Sent: Thursday, May 2, 2013 7:49:53 PM<br>
| Subject: Re: [infinispan-internal] Message flow tracer/analyzer<br>
|<br>
| Did you look into Twitter's Zipkin? It looks very suited for doing this kind<br>
| of stuff:<br>
| <a href="http://engineering.twitter.com/2012/06/distributed-systems-tracing-with-zipkin.html" target="_blank">http://engineering.twitter.com/2012/06/distributed-systems-tracing-with-zipkin.html</a><br>
|<br>
| It could be used in combination with Byteman and it gets you a nice user web<br>
| interface :)<br>
<br>
Thanks for pointing me to that, Galder, I should probably read more blogs :)<br>
<br>
This looks very similar and integrating with the nice GUI could be cool, but I am not sure whether it's worth the effort (read: I probably won't have time for this).<br>
The main difference is that with Byteman I don't have to modify the code to pass the zipkin header along with the request. Not relying on Byteman would be cool (I don't believe that the few accesses of ConcurrentHashMap and ThreadLocal variables slow the system 3 times, Byteman must introduce a lot of its own overhead). Eventually I may end up with interpreting the rules on JGroups/ISPN source level and inserting the calls directly into the source code. I know Byteman can do that on the bytecode level but there are some bugs that prohibit from doing so (and Andrew Dinn noted that these may end up being "won't fix" as fixing them may break other stuff).<br>
<br>
Nevertheless, I will probably rename data -> annotation, control flow -> span, message flow -> trace in order to be consistent with Zipkin naming.<br>
<br>
|<br>
| p.s. It's written in Scala.<br>
<br>
Oh, <irony>great!</irony> ;-)<br>
<br>
Radim<br>
<br>
<br>
|<br>
| On May 2, 2013, at 2:48 PM, Radim Vansa <<a href="mailto:rvansa@redhat.com">rvansa@redhat.com</a>> wrote:<br>
|<br>
| > Good news, everyone,<br>
| ><br>
| > in the last two weeks I've been working on a tool that could help us to<br>
| > profile Infinispan performance, analyze it and probably debug some stuff<br>
| > as well. While trace logs are the most useful, performance is impacted to<br>
| > almost unusable levels and it still does not provide enough information,<br>
| > logs have low precision etc.<br>
| > The idea is to analyze the behaviour based on the requests and track down<br>
| > the consequences of each request (put/get/whatever). Currently I have a<br>
| > working prototype (I believe already useful) which is able to track down<br>
| > all messages based on the initial request, records which threads execute<br>
| > etc. It's Byteman based, no trace logs/code changes required. However,<br>
| > according to my initial testing it reduces the overall performance 2-3<br>
| > times.<br>
| ><br>
| > The code is located in <a href="https://github.com/rvansa/message-flow-tracer" target="_blank">https://github.com/rvansa/message-flow-tracer</a> ,<br>
| > please look at README for details what it can do and ping me if you have<br>
| > any questions/feedback.<br>
| ><br>
| > Radim<br>
| ><br>
| > PS: short demo output on <a href="http://pastebin.com/raw.php?i=SBQFuG3a" target="_blank">http://pastebin.com/raw.php?i=SBQFuG3a</a><br>
| ><br>
| > -----------------------------------------------------------<br>
| > Radim Vansa<br>
| > Quality Assurance Engineer<br>
| > JBoss Datagrid<br>
| > tel. <a href="tel:%2B420532294559%20ext.%2062559" value="+420532294559">+420532294559 ext. 62559</a><br>
| ><br>
| > Red Hat Czech, s.r.o.<br>
| > Brno, Purkyňova 99/71, PSČ 612 45<br>
| > Czech Republic<br>
| ><br>
| ><br>
|<br>
|<br>
| --<br>
| Galder Zamarreño<br>
| <a href="mailto:galder@redhat.com">galder@redhat.com</a><br>
| <a href="http://twitter.com/galderz" target="_blank">twitter.com/galderz</a><br>
|<br>
| Project Lead, Escalante<br>
| <a href="http://escalante.io" target="_blank">http://escalante.io</a><br>
|<br>
| Engineer, Infinispan<br>
| <a href="http://infinispan.org" target="_blank">http://infinispan.org</a><br>
|<br>
|<br>
<br>
_______________________________________________<br>
infinispan-dev mailing list<br>
<a href="mailto:infinispan-dev@lists.jboss.org">infinispan-dev@lists.jboss.org</a><br>
<a href="https://lists.jboss.org/mailman/listinfo/infinispan-dev" target="_blank">https://lists.jboss.org/mailman/listinfo/infinispan-dev</a></blockquote></div>