As an experiment I'm instrumenting a service written in clojure using opentracing-java. Through the clojure/java interop I've mostly succeeded in getting trace information reported through to the Hawkular APM server.
I say "mostly succeeded" because sooner or later in every one of my hacking sessions I get to the point where the spans I am creating in the app are no longer reported in the web ui.
For convenience I'm using the Hawkular dev
docker image. In my test app I'm doing nothing more than initializing an
APMTracer with the appropriate environment variables set, and then calling buildSpan("foo"), withTag("sampling.priority", 1), start(), sleep for a while, and then finish(). Where all of the previous was done in clojure, but I'm talking in pseudocode here just to make the intent clear.
So like I said, sometimes these traces are reported, other times they seem to be silently dropped. I can't detect any consistent pattern how or why this happens...
(1) Is using a "sampling.priority" of 1 merely advisory? It would explain everything if those traces are meant to be dropped.
(2) Is there any convenient way I can see, with increased logging or something, which traces are actually being sent from the client, and which are actually received by the server?