7:31 p.m.
As an experiment I'm instrumenting a service written in clojure using
opentracing-java. Through the clojure/java interop I've mostly succeeded
in getting trace information reported through to the Hawkular APM server.
I say "mostly succeeded" because sooner or later in every one of my hacking
sessions I get to the point where the spans I am creating in the app are no
longer reported in the web ui.
For convenience I'm using the Hawkular dev docker image
<https://hub.docker.com/r/jboss/hawkular-apm-server-dev>;. In my test app
I'm doing nothing more than initializing an APMTracer
<https://github.com/hawkular/hawkular-apm/blob/master/client/opentracing/s...
with
the appropriate environment variables set, and then calling
buildSpan("foo"), withTag("sampling.priority", 1), start(), sleep for
a
while, and then finish(). Where all of the previous was done in clojure,
but I'm talking in pseudocode here just to make the intent clear.
So like I said, sometimes these traces are reported, other times they seem
to be silently dropped. I can't detect any consistent pattern how or why
this happens...
(1) Is using a "sampling.priority" of 1 merely advisory? It would explain
everything if those traces are meant to be dropped.
(2) Is there any convenient way I can see, with increased logging or
something, which traces are actually being sent from the client, and which
are actually received by the server?
3:54 a.m.
Hi Neil
----- Original Message -----
As an experiment I'm instrumenting a service written in clojure using
opentracing-java. Through the clojure/java interop I've mostly succeeded in
getting trace information reported through to the Hawkular APM server.
I say "mostly succeeded" because sooner or later in every one of my hacking
sessions I get to the point where the spans I am creating in the app are no
longer reported in the web ui.
For convenience I'm using the Hawkular dev docker image . In my test app I'm
doing nothing more than initializing an APMTracer with the appropriate
environment variables set, and then calling buildSpan("foo"),
withTag("sampling.priority", 1), start(), sleep for a while, and then
finish(). Where all of the previous was done in clojure, but I'm talking in
pseudocode here just to make the intent clear.
So like I said, sometimes these traces are reported, other times they seem to
be silently dropped. I can't detect any consistent pattern how or why this
happens...
(1) Is using a "sampling.priority" of 1 merely advisory? It would explain
everything if those traces are meant to be dropped.
If using the default constructor for APMTracer, then the default behaviour should be to
trace all - and setting the sampling.priority to 1 should not override that. Could you try
not setting this tag to see if there is any difference?
(2) Is there any convenient way I can see, with increased logging or
something, which traces are actually being sent from the client, and which
are actually received by the server?
You could initially check the traces stored in Elasticsearch using something like: curl
http://localhost:9200/apm-hawkular/trace/_search | python -m json.tool
Do you have a pure Java example that reproduces the same issue? Might be worth creating a
jira in https://issues.jboss.org/projects/HWKAPM to track the issue.
Regards
Gary
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
8:39 a.m.
Thanks Gary.
On Fri, Feb 3, 2017 at 1:54 AM, Gary Brown <gbrown(a)redhat.com> wrote:
> (1) Is using a "sampling.priority" of 1 merely advisory? It would explain
> everything if those traces are meant to be dropped.
If using the default constructor for APMTracer, then the default behaviour
should be to trace all - and setting the sampling.priority to 1 should not
override that. Could you try not setting this tag to see if there is any
difference?
I see. Well, I am using the default constructor, and I have tried with and
without sampling.priority=1 and it's the same situation either way.
(2) Is there any convenient way I can see, with increased logging or
> something, which traces are actually being sent from the client, and
which
> are actually received by the server?
You could initially check the traces stored in Elasticsearch using
something like: curl http://localhost:9200/apm-hawkular/trace/_search |
python -m json.tool
Right now I have a repl launched with HAWKULAR_APM_LOG_LEVEL set to
FINEST. I'm creating spans in the repl as described earlier. Each time I
create a trace I see a log entry from the client like this:
FINEST: [TracePublisherRESTClient] [Thread[pool-2-thread-1,5,main]]
Status code is: 204
and that 204 would suggest the trace info was successfully sent. But
inside the docker container I can curl Elasticsearch and those new traces
are not to be found.
Incidentally, I started the repl last night, did a few successful tests,
and then closed the lid of my laptop for the night with the Hawkular
container still running and the repl still running. I've also had this
issue occur immediately on launch of the repl, so I don't think it's
specifically about long running repls and/or sleeping, but for completeness
I thought I would clarify how I am running this.
Do you have a pure Java example that reproduces the same issue? Might
be
worth creating a jira in https://issues.jboss.org/projects/HWKAPM to track
the issue.
No, not yet...
9:15 a.m.
Hi Neil
Sounds strange. Would it be possible to try running the server outside docker to see if
there may be issues there.
If you create the jira with reproducer then we will investigate aswell.
Thanks for the additional info.
Regards
Gary
----- Original Message -----
Thanks Gary.
On Fri, Feb 3, 2017 at 1:54 AM, Gary Brown < gbrown(a)redhat.com > wrote:
> (1) Is using a "sampling.priority" of 1 merely advisory? It would explain
> everything if those traces are meant to be dropped.
If using the default constructor for APMTracer, then the default behaviour
should be to trace all - and setting the sampling.priority to 1 should not
override that. Could you try not setting this tag to see if there is any
difference?
I see. Well, I am using the default constructor, and I have tried with and
without sampling.priority=1 and it's the same situation either way.
> (2) Is there any convenient way I can see, with increased logging or
> something, which traces are actually being sent from the client, and which
> are actually received by the server?
You could initially check the traces stored in Elasticsearch using something
like: curl http://localhost:9200/apm-hawkular/trace/_search | python -m
json.tool
Right now I have a repl launched with HAWKULAR_APM_LOG_LEVEL set to FINEST.
I'm creating spans in the repl as described earlier. Each time I create a
trace I see a log entry from the client like this:
FINEST: [TracePublisherRESTClient] [Thread[pool-2-thread-1,5,main]] Status
code is: 204
and that 204 would suggest the trace info was successfully sent. But inside
the docker container I can curl Elasticsearch and those new traces are not
to be found.
Incidentally, I started the repl last night, did a few successful tests, and
then closed the lid of my laptop for the night with the Hawkular container
still running and the repl still running. I've also had this issue occur
immediately on launch of the repl, so I don't think it's specifically about
long running repls and/or sleeping, but for completeness I thought I would
clarify how I am running this.
> Do you have a pure Java example that reproduces the same issue? Might be
> worth creating a jira in https://issues.jboss.org/projects/HWKAPM to track
> the issue.
No, not yet...
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
9:53 a.m.
Thanks Gary. I'll try running the server outside docker, but before I do that I'm
going to run the container on a machine with more memory.
On Feb 3, 2017, at 7:15 AM, Gary Brown <gbrown(a)redhat.com>
wrote:
Hi Neil
Sounds strange. Would it be possible to try running the server outside docker to see if
there may be issues there.
If you create the jira with reproducer then we will investigate aswell.
Thanks for the additional info.
Regards
Gary
----- Original Message -----
> Thanks Gary.
>
> On Fri, Feb 3, 2017 at 1:54 AM, Gary Brown < gbrown(a)redhat.com > wrote:
>
>
>
>> (1) Is using a "sampling.priority" of 1 merely advisory? It would
explain
>> everything if those traces are meant to be dropped.
>
> If using the default constructor for APMTracer, then the default behaviour
> should be to trace all - and setting the sampling.priority to 1 should not
> override that. Could you try not setting this tag to see if there is any
> difference?
>
> I see. Well, I am using the default constructor, and I have tried with and
> without sampling.priority=1 and it's the same situation either way.
>
>
>
>> (2) Is there any convenient way I can see, with increased logging or
>> something, which traces are actually being sent from the client, and which
>> are actually received by the server?
>
> You could initially check the traces stored in Elasticsearch using something
> like: curl http://localhost:9200/apm-hawkular/trace/_search | python -m
> json.tool
>
> Right now I have a repl launched with HAWKULAR_APM_LOG_LEVEL set to FINEST.
> I'm creating spans in the repl as described earlier. Each time I create a
> trace I see a log entry from the client like this:
>
> FINEST: [TracePublisherRESTClient] [Thread[pool-2-thread-1,5,main]] Status
> code is: 204
>
> and that 204 would suggest the trace info was successfully sent. But inside
> the docker container I can curl Elasticsearch and those new traces are not
> to be found.
>
> Incidentally, I started the repl last night, did a few successful tests, and
> then closed the lid of my laptop for the night with the Hawkular container
> still running and the repl still running. I've also had this issue occur
> immediately on launch of the repl, so I don't think it's specifically about
> long running repls and/or sleeping, but for completeness I thought I would
> clarify how I am running this.
>
>> Do you have a pure Java example that reproduces the same issue? Might be
>> worth creating a jira in https://issues.jboss.org/projects/HWKAPM to track
>> the issue.
>
> No, not yet...
>
> _______________________________________________
> hawkular-dev mailing list
> hawkular-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hawkular-dev
>
_______________________________________________
hawkular-dev mailing list
hawkular-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hawkular-dev
2850
days inactive
2850
days old
4 comments
3 participants
participants (3)
-
Gary Brown -
Neil Okamoto -
Neil Okamoto