<div dir="ltr"><div>Since this morning I've had a server running inside docker on a separate machine with more installed memory. I haven't seen any problems since then. In retrospect I wish I had thought of this sooner.</div><div><br></div><div>For now I'm moving on from this problem to do a more complete instrumentation of the clojure app. I'll keep an eye open for further problems and I'll report back if there's anything noteworthy.</div><div><div class="gmail_extra"><br></div><div class="gmail_extra">thanks Gary,</div><div class="gmail_extra">Neil</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 3, 2017 at 7:53 AM, Neil Okamoto <span dir="ltr"><<a href="mailto:neil.okamoto@gmail.com" target="_blank">neil.okamoto@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Thanks Gary. I'll try running the server outside docker, but before I do that I'm going to run the container on a machine with more memory.<br>
<div class="gmail-HOEnZb"><div class="gmail-h5"><br>
> On Feb 3, 2017, at 7:15 AM, Gary Brown <<a href="mailto:gbrown@redhat.com">gbrown@redhat.com</a>> wrote:<br>
><br>
> Hi Neil<br>
><br>
> Sounds strange. Would it be possible to try running the server outside docker to see if there may be issues there.<br>
><br>
> If you create the jira with reproducer then we will investigate aswell.<br>
><br>
> Thanks for the additional info.<br>
><br>
> Regards<br>
> Gary<br>
><br>
> ----- Original Message -----<br>
>> Thanks Gary.<br>
>><br>
>> On Fri, Feb 3, 2017 at 1:54 AM, Gary Brown < <a href="mailto:gbrown@redhat.com">gbrown@redhat.com</a> > wrote:<br>
>><br>
>><br>
>><br>
>>> (1) Is using a "sampling.priority" of 1 merely advisory? It would explain<br>
>>> everything if those traces are meant to be dropped.<br>
>><br>
>> If using the default constructor for APMTracer, then the default behaviour<br>
>> should be to trace all - and setting the sampling.priority to 1 should not<br>
>> override that. Could you try not setting this tag to see if there is any<br>
>> difference?<br>
>><br>
>> I see. Well, I am using the default constructor, and I have tried with and<br>
>> without sampling.priority=1 and it's the same situation either way.<br>
>><br>
>><br>
>><br>
>>> (2) Is there any convenient way I can see, with increased logging or<br>
>>> something, which traces are actually being sent from the client, and which<br>
>>> are actually received by the server?<br>
>><br>
>> You could initially check the traces stored in Elasticsearch using something<br>
>> like: curl <a href="http://localhost:9200/apm-hawkular/trace/_search" rel="noreferrer" target="_blank">http://localhost:9200/apm-<wbr>hawkular/trace/_search</a> | python -m<br>
>> json.tool<br>
>><br>
>> Right now I have a repl launched with HAWKULAR_APM_LOG_LEVEL set to FINEST.<br>
>> I'm creating spans in the repl as described earlier. Each time I create a<br>
>> trace I see a log entry from the client like this:<br>
>><br>
>> FINEST: [TracePublisherRESTClient] [Thread[pool-2-thread-1,5,<wbr>main]] Status<br>
>> code is: 204<br>
>><br>
>> and that 204 would suggest the trace info was successfully sent. But inside<br>
>> the docker container I can curl Elasticsearch and those new traces are not<br>
>> to be found.<br>
>><br>
>> Incidentally, I started the repl last night, did a few successful tests, and<br>
>> then closed the lid of my laptop for the night with the Hawkular container<br>
>> still running and the repl still running. I've also had this issue occur<br>
>> immediately on launch of the repl, so I don't think it's specifically about<br>
>> long running repls and/or sleeping, but for completeness I thought I would<br>
>> clarify how I am running this.<br>
>><br>
>>> Do you have a pure Java example that reproduces the same issue? Might be<br>
>>> worth creating a jira in <a href="https://issues.jboss.org/projects/HWKAPM" rel="noreferrer" target="_blank">https://issues.jboss.org/<wbr>projects/HWKAPM</a> to track<br>
>>> the issue.<br>
>><br>
>> No, not yet...<br>
>><br>
>> ______________________________<wbr>_________________<br>
>> hawkular-dev mailing list<br>
>> <a href="mailto:hawkular-dev@lists.jboss.org">hawkular-dev@lists.jboss.org</a><br>
>> <a href="https://lists.jboss.org/mailman/listinfo/hawkular-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/hawkular-dev</a><br>
>><br>
> ______________________________<wbr>_________________<br>
> hawkular-dev mailing list<br>
> <a href="mailto:hawkular-dev@lists.jboss.org">hawkular-dev@lists.jboss.org</a><br>
> <a href="https://lists.jboss.org/mailman/listinfo/hawkular-dev" rel="noreferrer" target="_blank">https://lists.jboss.org/<wbr>mailman/listinfo/hawkular-dev</a><br>
</div></div></blockquote></div><br></div></div></div>