2:47 p.m.
Executive Summary:
- We need to update to at least OpenTelemetry 1.62.0 to avoid a CVE in Otel
- That update brings in code changes that break when running under a Security Manager
- Proposed solutions:
- Replace BatchSpanProcessor with custom impl/fork
- Use SimpleSpanProcessor when SM is detected
Details:
When running the proposed OpenTelemetry update (1.48.0 -> 1.62.0), we run into Security
Manger issues in code introduced since the 1.48.0 release. Some of these failures can be
worked around by changes in SmallRye OpenTelemetry (see
https://gist.github.com/jasondlee/83906a7b6aad2336dad22377d380fc0c). That feels a bit
clumsy, but is only necessary (for us) until WildFly removes SM support altogether.
Regardless, that leaves one final failure, which we're seeing in 6 places in our test
suite (NB: the MP Telemetry passes without complaint). The code in our test suite that
causes the failure is this:
final Span span = tracer.spanBuilder("Saying hello from service
1").startSpan();
try (Scope scope = span.makeCurrent()) {
span.addEvent("some-event");
span.setAttribute("name", name);
span.end();
}
which is, I believe, a perfectly valid use of the APIs.
The problem occurs in the BatchSpanProcessor provided by opentelemetry-java. This class
exports spans in batches, and does so via private static final class Worker. In the
constructor for this class, an instance of SpanProcessorInstrumentation is manually
created, with no apparent way to control or configure externally. The actual instance of
that class is either LegacySpanProcessorInstrumentation or
SemConvSpanProcessorInstrumentation, but both classes have the same issue, which is this:
RESTEASY002375: Error processing request GET /_DEFAULT__service-one_service-one -
org.wildfly.test.integration.observability.opentelemetry.application.OtelService1.sayHello:
java.security.AccessControlException:
WFSM000001: Permission check failed (permission
"("java.lang.RuntimePermission" "getClassLoader")" in code
source
"(vfs:/content/_DEFAULT__service-one_service-one.war/WEB-INF/classes <no signer
certificates>)" of "ModuleClassLoader for Module
"deployment._DEFAULT__service-one_service-one.war" from Service Module
Loader")
at
org.wildfly.security.elytron-base@2.9.1.Final//org.wildfly.security.manager.WildFlySecurityManager.checkPermission(WildFlySecurityManager.java:341)
at
org.wildfly.security.elytron-base@2.9.1.Final//org.wildfly.security.manager.WildFlySecurityManager.checkPermission(WildFlySecurityManager.java:233)
at java.base/java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2060)
at java.base/java.lang.Thread.getContextClassLoader(Thread.java:1502)
at
io.opentelemetry.sdk@1.62.0//io.opentelemetry.sdk.metrics.internal.state.CallbackRegistration.<init>(CallbackRegistration.java:39)
at
io.opentelemetry.sdk@1.62.0//io.opentelemetry.sdk.metrics.internal.state.CallbackRegistration.create(CallbackRegistration.java:68)
at
io.opentelemetry.sdk@1.62.0//io.opentelemetry.sdk.metrics.InstrumentBuilder.buildLongAsynchronousInstrument(InstrumentBuilder.java:89)
at
io.opentelemetry.sdk@1.62.0//io.opentelemetry.sdk.metrics.SdkLongGauge$SdkLongGaugeBuilder.buildWithCallback(SdkLongGauge.java:86)
at
io.opentelemetry.sdk@1.62.0//io.opentelemetry.sdk.trace.export.LegacySpanProcessorInstrumentation.buildQueueMetricsOnce(LegacySpanProcessorInstrumentation.java:76)
Both classes create a lambda to be used as a callback, and both handle that callback using
CallbackRegistration, which falls overunder SM with this line: this.contextClassLoader =
Thread.currentThread().getContextClassLoader(); I have found no way to avoid this code
path using the public APIs.
One solution I can see -- and this is where I'd love some feedback from clear, extra
eyes -- is to provide our own SpanProcessor -- even if temporarily -- that avoids using
any SpanProcessorInstrumentation. As best as I can tell, this class only adds metrics
related to the SpanProcessor itself (e.g., queueSize), which I doubt is mission critical
(though, if it is, with a custom processor, we can provide that information in a
SM-friendly manner).
Another option is to use the SimpleSpanProcessor. Rather than sending spans in batches,
this will publish each span as it is closed. I've not maintained an OpenTelemetry
collector and/or an observability dashboard in production, so I don't know what the
implications of that might be, but the users will see different behavior, if still
technically the same functionality. Or so.
I've tested using SimpleSpanProcessor locally, and it seems to work fine (though it
did find even more SM-related issues in the SmallRye OpenTelemetry codebase, which
I've addressed). If we go this route, we could simple use the default
BatchSpanProcessor unless we detect the Security Manager, in which case we could switch to
the SimpleSpanProcessor. That would leave non-SM scenarios unchanged, and only a slight,
hopefully acceptable change in behavior under SM. Or we can go with the full-throated
custom, batched SpanProcessor impl, which maintains behavior, but, of course, new code
means the possibility of new bugs.
I'm open to suggestions and enhancements. And, if you've read this far, my thanks
to you. :)
Jason Lee
OKC JUG President
https://jasondl.ee
https://twitter.com/jasondlee
http://linkedin.com/in/jasondlee