Since this morning I've had a server running inside docker on a separate machine with more installed memory. I haven't seen any problems since then. In retrospect I wish I had thought of this sooner.

For now I'm moving on from this problem to do a more complete instrumentation of the clojure app. I'll keep an eye open for further problems and I'll report back if there's anything noteworthy.

thanks Gary,

Neil

On Fri, Feb 3, 2017 at 7:53 AM, Neil Okamoto <neil.okamoto@gmail.com> wrote:

Thanks Gary. I'll try running the server outside docker, but before I do that I'm going to run the container on a machine with more memory.

> On Feb 3, 2017, at 7:15 AM, Gary Brown <gbrown@redhat.com> wrote:
>
> Hi Neil
>
> Sounds strange. Would it be possible to try running the server outside docker to see if there may be issues there.
>
> If you create the jira with reproducer then we will investigate aswell.
>
> Thanks for the additional info.
>
> Regards
> Gary
>
> ----- Original Message -----
>> Thanks Gary.
>>
>> On Fri, Feb 3, 2017 at 1:54 AM, Gary Brown < gbrown@redhat.com > wrote:
>>
>>
>>
>>> (1) Is using a "sampling.priority" of 1 merely advisory? It would explain
>>> everything if those traces are meant to be dropped.
>>
>> If using the default constructor for APMTracer, then the default behaviour
>> should be to trace all - and setting the sampling.priority to 1 should not
>> override that. Could you try not setting this tag to see if there is any
>> difference?
>>
>> I see. Well, I am using the default constructor, and I have tried with and
>> without sampling.priority=1 and it's the same situation either way.
>>
>>
>>
>>> (2) Is there any convenient way I can see, with increased logging or
>>> something, which traces are actually being sent from the client, and which
>>> are actually received by the server?
>>
>> You could initially check the traces stored in Elasticsearch using something
>> like: curl http://localhost:9200/apm-hawkular/trace/_search | python -m
>> json.tool
>>
>> Right now I have a repl launched with HAWKULAR_APM_LOG_LEVEL set to FINEST.
>> I'm creating spans in the repl as described earlier. Each time I create a
>> trace I see a log entry from the client like this:
>>
>> FINEST: [TracePublisherRESTClient] [Thread[pool-2-thread-1,5,main]] Status
>> code is: 204
>>
>> and that 204 would suggest the trace info was successfully sent. But inside
>> the docker container I can curl Elasticsearch and those new traces are not
>> to be found.
>>
>> Incidentally, I started the repl last night, did a few successful tests, and
>> then closed the lid of my laptop for the night with the Hawkular container
>> still running and the repl still running. I've also had this issue occur
>> immediately on launch of the repl, so I don't think it's specifically about
>> long running repls and/or sleeping, but for completeness I thought I would
>> clarify how I am running this.
>>
>>> Do you have a pure Java example that reproduces the same issue? Might be
>>> worth creating a jira in https://issues.jboss.org/projects/HWKAPM to track
>>> the issue.
>>
>> No, not yet...
>>
>> _______________________________________________
>> hawkular-dev mailing list
>> hawkular-dev@lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev@lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev