Executive Summary:
- We need to update to at least OpenTelemetry 1.62.0 to avoid a CVE in Otel
- That update brings in code changes that break when running under a Security Manager
- Proposed solutions:
- Replace BatchSpanProcessor with custom impl/fork
- Use SimpleSpanProcessor when SM is detected
Details:
When running the proposed OpenTelemetry update (1.48.0 -> 1.62.0), we run into Security Manger issues in code introduced since the 1.48.0 release. Some of these failures can be worked around by changes in SmallRye OpenTelemetry (see https://gist.github.com/jasondlee/83906a7b6aad2336dad22377d380fc0c). That feels a bit clumsy, but is only necessary (for us) until WildFly removes SM support altogether. Regardless, that leaves one final failure, which we're seeing in 6 places in our test suite (NB: the MP Telemetry passes without complaint). The code in our test suite that causes the failure is this:
final Span span = tracer.spanBuilder("Saying hello from service 1").startSpan();
try (Scope scope = span.makeCurrent()) {
span.addEvent("some-event");
span.setAttribute("name", name);
span.end();
}
which is, I believe, a perfectly valid use of the APIs.
The problem occurs in the BatchSpanProcessor provided by opentelemetry-java. This class exports spans in batches, and does so via private static final class Worker. In the constructor for this class, an instance of SpanProcessorInstrumentation is manually created, with no apparent way to control or configure externally. The actual instance of that class is either LegacySpanProcessorInstrumentation or SemConvSpanProcessorInstrumentation, but both classes have the same issue, which is this:
RESTEASY002375: Error processing request GET /_DEFAULT__service-one_service-one -
org.wildfly.test.integration.observability.opentelemetry.application.OtelService1.sayHello: java.security.AccessControlException:
WFSM000001: Permission check failed (permission "("java.lang.RuntimePermission" "getClassLoader")" in code source
"(vfs:/content/_DEFAULT__service-one_service-one.war/WEB-INF/classes <no signer certificates>)" of "ModuleClassLoader for Module
"deployment._DEFAULT__service-one_service-one.war" from Service Module Loader")
at org.wildfly.security.elytron-base@2.9.1.Final//org.wildfly.security.manager.WildFlySecurityManager.checkPermission(WildFlySecurityManager.java:341)
at org.wildfly.security.elytron-base@2.9.1.Final//org.wildfly.security.manager.WildFlySecurityManager.checkPermission(WildFlySecurityManager.java:233)
at java.base/java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2060)
at java.base/java.lang.Thread.getContextClassLoader(Thread.java:1502)
at io.opentelemetry.sdk@1.62.0//io.opentelemetry.sdk.metrics.internal.state.CallbackRegistration.<init>(CallbackRegistration.java:39)
at io.opentelemetry.sdk@1.62.0//io.opentelemetry.sdk.metrics.internal.state.CallbackRegistration.create(CallbackRegistration.java:68)
at io.opentelemetry.sdk@1.62.0//io.opentelemetry.sdk.metrics.InstrumentBuilder.buildLongAsynchronousInstrument(InstrumentBuilder.java:89)
at io.opentelemetry.sdk@1.62.0//io.opentelemetry.sdk.metrics.SdkLongGauge$SdkLongGaugeBuilder.buildWithCallback(SdkLongGauge.java:86)
at io.opentelemetry.sdk@1.62.0//io.opentelemetry.sdk.trace.export.LegacySpanProcessorInstrumentation.buildQueueMetricsOnce(LegacySpanProcessorInstrumentation.java:76)
Both classes create a lambda to be used as a callback, and both handle that callback using CallbackRegistration, which falls overunder SM with this line: this.contextClassLoader = Thread.currentThread().getContextClassLoader(); I have found no way to avoid this code path using the public APIs.
One solution I can see -- and this is where I'd love some feedback from clear, extra eyes -- is to provide our own SpanProcessor -- even if temporarily -- that avoids using any SpanProcessorInstrumentation. As best as I can tell, this class only adds metrics related to the SpanProcessor itself (e.g., queueSize), which I doubt is mission critical (though, if it is, with a custom processor, we can provide that information in a SM-friendly manner).
Another option is to use the SimpleSpanProcessor. Rather than sending spans in batches, this will publish each span as it is closed. I've not maintained an OpenTelemetry collector and/or an observability dashboard in production, so I don't know what the implications of that might be, but the users will see different behavior, if still technically the same functionality. Or so.
I've tested using SimpleSpanProcessor locally, and it seems to work fine (though it did find even more SM-related issues in the SmallRye OpenTelemetry codebase, which I've addressed). If we go this route, we could simple use the default BatchSpanProcessor unless we detect the Security Manager, in which case we could switch to the SimpleSpanProcessor. That would leave non-SM scenarios unchanged, and only a slight, hopefully acceptable change in behavior under SM. Or we can go with the full-throated custom, batched SpanProcessor impl, which maintains behavior, but, of course, new code means the possibility of new bugs.
I'm open to suggestions and enhancements. And, if you've read this far, my thanks to you. :)