[wildfly-dev] Thoughts on alternative security managers

[David M. Lloyd]

It is possible to implement a security manager which behaves differently 
than the default AccessController-based security manager.  I am 
wondering if we should maybe explore some options here.

AccessControlContext (ACC) permission checks are expensive.  The 
assembled context can consist of many object instances, many of which 
may be constructed on the fly, in order to check a permission against 
every invocation frame on the call stack (including going through thread 
start points to earlier, snapshotted thread stacks) down to the most 
recent doPrivileged() invocation.  The theory is to ensure that 
permissions can never escalate without an explicit action.  However, few 
frameworks use doPrivileged() properly, resulting in too many 
permissions being assigned to the user deployment anyway.

Option 0: Status quo
--------------------
To mitigate some of this cost, our current security manager has 
"checked" and "unchecked" modes for the current thread.  When running 
trusted code, we switch to "unchecked" mode which is functionally 
similar to a privileged block, except (a) it is much faster as expensive 
permission checks are simply bypassed and (b) it is the responsibility 
of the container to ensure that checked mode is re-enabled before 
calling back in to user code.

When checked mode is enabled, all permission checks happen in the usual 
way with ACC.  Privileged blocks are necessary for code that runs in 
checked mode (i.e. user libraries and many of our APIs that do not have 
direct support for WildFlySecurityManager).

Option 1: Fast and simple
-------------------------
We could completely switch the SecurityManager's (SM's) security context 
object to be a per-deployment context of some sort.  Permissions would 
be checked based on whatever deployment is currently active; checked and 
unchecked modes would still be used in this case.

In this option, the permissions that are currently checked are *always* 
the active deployment, which is particularly relevant in the case that 
the deployment calls into some other deployment without using container 
facilities.

We could still support standard security policies to assign additional 
permissions to the current Principal.

Privileged blocks in this case would be completely ignored.

This could be a very fast approach as no objects need be constructed to 
perform a permission check; however, it is also perhaps harder to create 
fine-grained restrictions on things.

Option 2: Call stack (simplified version)
-----------------------------------------
Alternatively, we could examine the call stack on each invocation to 
locate the most recent enclosing "untrusted" class loader.  This 
involves acquiring and traversing the invocation call stack (which may 
be cheaper in Java 8 or 9).  While expensive, this operation should be 
less expensive than constructing an ACC.

Privileged blocks in this case would also be completely ignored, and all 
the other above benefits would hold.  For the relative performance cost 
we would pay over Option 1, we'd regain the ability to establish 
fine-grained restrictions.

Option 3: Call stack (advanced version)
---------------------------------------
In this option we essentially duplicate what ACC does, but using the 
presumably less-expensive SM method of acquiring and examining the call 
stack.  Privileged blocks probably would not work, but if not, we'd have 
an alternative mechanism for establishing bounds on the set of 
permissions to check per call.  We would still support checked/unchecked 
mode.

All three of these options would not be compatible with DomainCombiners, 
if that matters.

Any thoughts?
-- 
- DML