[wildfly-dev] Proposal to add notifications to WildFly management model and API

Thu Jul 10 05:52:04 EDT 2014

On 8 Jul 2014, at 23:07, Brian Stansberry <brian.stansberry at redhat.com> wrote:

> On 7/7/14, 9:46 AM, Jeff Mesnil wrote:
>> # Add Notification support to WildFly Management
>> 
>> Tracked by https://issues.jboss.org/browse/WFLY-266
>> 
>> Use Cases
>> ---------
>> 
>> Notifications are an useful mechanism to observe management changes on WildFly servers.
>> It allows an administrator to be informed of changes outside of his own actions (e.g. a server has been killed, a new application is deployed, etc.)
>> 
>> Currently WildFly lacks notifications and users that were depending on JMX notifications in previous versions have no similar feature to use.
>> 
>> The most expected use cases for WildFly notifications are:
>> - enhance UX for Web console. Using notifications, the Web console could notify the users of changes outside its own actions.
>> - replacement for JMX notifications. Users that were listening for JMX notifications to observe management changes would have a similar feature using WildFly own notifications
>> - integration with JMX. Notifications emitted by WildFly could be converted and made available using JMX notifications (including notifications for mbean registered/unregistered)
>> 
>> Part 1: Notification Definition
>> -------------------------------
>> 
>> A resource will define the notifications it emits. These definitions will be added to the attributes and operations definitions on a resource.
>> 
>>       {
>> 	  "description" => "A manageable resource",
>>           "attributes" => {
>> 	     ...
>>           },
>> 	  "operations" => {
>> 	     ...
>>           },
>> 	  "notifications" => {
>> 	     "resource-added" => {
>> 	        ...
>> 	     }
>> 	  },
>> 	  "children" => {
>> 	     ...
>>           }
>>        }
>> 
>> The description of a notification will be composed of:
>> 
>> * type - String - the type of notification (resource-added, server-stopped, etc.)
>> * description - String - i18ned description of the notification
>> * access-constraints - the RBAC access constraints that controls who can receive the notifications
>> * data-type - ModelType or complex structure - optional - only present if the notification will have a data value. data-type will detail the structure of the data value, enumerating the value's fields and the type of their value
>> 
>> The read-resource-description will be enhanced with a notifications parameter (boolean) to include the notifications descriptions (default value is false, same as the operations parameter).
>> 
>> The ManagementResourceRegistration interface will be enhanced to register a notification definition with registerNotification(NotificationDefinition notification). The NotificationDefinition interface corresponds to the detyped representation of a notifications and comes with a builder API.
>> 
>> Part 2: Emitting a notification
>> -------------------------------
>> 
>> A notification can be emitted in any OperationStepHandler using the OperationContext.emit(Notification method)
>> 
>>     public void execute(OperationContext context, ModelNode operation) throws OperationFailedException {
>> 
>>        // perform some actions
>>        ...
>> 
>>        context.emit(new Notification(SERVER_RESTARTED_NOTIFICATION, address, ROOT_LOGGER.serverHasBeenRestarted()));
>>        context.stepCompleted();
>>     }
>> 
>> The notification is *not* emitted (i.e. delivered to interested parties) when OperationContext.emit() is called. It is emitted at the end of the operation step only if it is successful. A call to OperationContext.emit() will have no effect if the operation is rolled back.
> 
> To clarify: I believe your intent was delivery is at the end of the 
> overall operation execution, when it commits, not at the end of the 
> "operation step". I mention this because an operation execution can 
> consist of many steps, with even a basic write involving 2 or 3.

You are right, my description was not correct. The notifications are all emitted at the end of the overall operation execution after it commits.
Either all notifications are emitted or none are.

> 
>> Notification emission is done asynchronously using the server thread pool and does not block the execution of the operation that triggered the notification: having zero or any notification handlers must have no impact of the execution of the operation.
>> 
>> A Notification is a simple Java class that represents the notification. It is composed of:
>> 
>> * type - String - the notification type
>> * address - PathAddress - the address of the resource that emits the notification
>> * message - String - the i18ned description of the message
>> * timestamp - long - the timestamp of the notification. It is set when the Notification object is created.
>> * data - ModelNode - optional - a detyped representation of data associated to the notification. If a notification includes a data field, its definition must describe it (in its data-type parameter).
>> 
>> If RBAC is enabled, the notification access-constraints will be checked to ensure that the handler have the required privileges to receive the notification. Notification will potentially contain critical information (e.g. if a security-credential attribute is updated, the notification will contain its old and new values) and must be constrained accordingly.
>> 
> 
> We also need to use SecurityManager permissions around the handler 
> registration. Non-remote registrations basically get all permissions, 
> same as internal management clients like the deployment-scanner do. We
> control the ability of internal management clients like the scanner to
> create a ModelControllerClient using SecurityManager permissions.
> 
>> Part 3: Global Resource Notifications
>> ——————————————————
>> 
>> In the same way that some operations are available for any resource (e.g. add, remove, read-resource-description), some notifications will be added to any resource of WildFly management model:
>> 
>> * resource-added - when a resource is added, it emits a resource-added notification
>> * resource-removed - when a resource is removed, it emits a resource-removed notification
>> * attribute-value-written - when a write-attribute operation is performed successfully on a resource, it emits a attribute-value-written    notification. The notification's data field contains the following information:
>>   * name - String - the name of the attribute
>>   * old-value - the detyped representation of the previous value of the attribute
>>   * new-value - the detyped representation of the new value
>> 
> 
>> 
>> Part 4: Notification Handlers
>> ——————————————
>> 
>> Any interested parties can receive notifications by registering a NotificationHandler using the ModelController.getNotificationSupport().registerNotificationHandler(source, handler, filter) method.
>> 
>> The source is a path address to handle notifications emitted by resources at this address.
>> The NotificationHandler is an interface with a single handleNotification(Notification notification) method.
>> The isNotificationEnabled(Notification notification) is an interface with a single isNotificationEnabled(Notification notification) method to filter out uninteresting notifications.
>> 
>> There is a similar unregister method to unregister a (handler, filter)
>> 
>> To be useful, the source path address will have to accept wildcards for the address' values:
>> * /subsystem=messaging/hornetq-server=* to receive notifications emitted by any hornetq-server resources
>> * /subsystem=messaging/hornetq-server=*/jms-queue=* to receive notifications emitted by any jms-queue on any hornetq-server resources
>> 
>> Wildcards for address' keys or key/value paris are not allowed (/subsystem=messaging/*=*/jms-queue=* and /subsystem=messaging/*/jms-queue=* are not valid).
>> 
>> This notion of wildcard for the resource addresses should be made to match current usage (e.g. in the CLI).
>> 
>> The main reason for the wildcard is for the resource-added/resource-removed notifications. I find more intuitive to have the notifications at the same resource-level than their corresponding add/remove operations.
> 
> Agreed. I don't think this should be an issue. We shouldn't use the 
> Resource tree anyway as the data structure for retaining the registered 
> handlers; a separate structure is needed. The wildcards are fine as long 
> as there's a relevant ManagementResourceRegistration.

In my POC, I am using a single Map to retain the handler and is separate from the resource tree.
The Map keys are the path address from the registerNotificationHandler() method (that can be wildcard addresses).
When a notification is emitted, I iterate on its entries to find the handlers that match the actual address of resource emitting the notification.
I.e. the data structure grows with the number of handlers, not with the resources that emits notifications.

>> However until the resource is created, there is no way to register a notification listener on it without using a wildcard.
>> If that proves problematic, we could change this approach with two alternatives:
>> * have a single well-known resource emit the notifications for all resource (that's the JMX approach). A likely candidate would be /core-service=management
>> * the resource-added/-removed notifications can be emitted by the resource parents (but it only fixes the issue for the last leaf of the address tree…)
>> 
>> I still have questions about RBAC enforcements and it is possible that the registration of a handler will have to be done with additional metadata identifying the user roles wrt RBAC...
>> 
> 
> It's not critical until you get to Part 7 but we'll need to sort it out 
> before even starting on Part 7.
> 
> The user-related inputs into the existing RBAC logic (see 
> org.jboss.as.controller.access.Authorizer) are the Caller and 
> Environment. The Caller basically just wraps a Subject, exposing the 
> data from the relevant Principal types (name and groups).
> 
> We could just cache the Caller in effect when the handler is registered 
> and use it for authorizing delivery of each notification. The problem is 
> if the user if no longer valid or the groups associated with the user 
> have changed, this won't be picked up.
> 
> We don't want to cache any "roles". First, the mapping of the Caller to 
> roles can change from time to time so we don't want to cache that 
> result. Second, "roles" are not first-class parts of the Authorizer API; 
> they are used by our standard impls of it, but we want to allow custom
> impls that may not care about roles.
> 
>> Part 5: Domain Notifications
>> ——————————————
>> 
>> Notifications are also intended to work in domain mode. In particular, they will be used to observe server state.
>> 
>> The following notifications will be emitted by resources at /host=XXX/server-config=YYY (i.e. the resource to start/stop/etc. a server):
>> * server-started
>> * server-stopped
>> * server-restarted
>> * server-destroyed
>> * server-killed
>> 
>> Part 6: Integration with local JMX
>> —————————————————
>> 
>> The jmx subsystem will be updated to leverage the WildFly notifications and expose them as MBean notifications in our jmx facade for the management model:
>> * the WildFly notification description will be converted to MBeanNotificationInfo and added to the MBeanInfo
>> * when a JMX notification listener is added to an ObjectName, a WildFly NotificationHandler will be added to the path address corresponding to the ObjectName.
>> * depending on the user feedback, we may provide a hack to convert some WildFly notifications to their well-known JMX equivalent notifications (e.g. resource-added => jmx.mbean.registered).
>> 
>> In a first step, integration will be limited to use of JMX locally. Remoting will not be supported.
>> 
> 
> Everything up to this point I'd like to see in 8.2.

I have create JIRA issues for the parts (as dependencies of the umbrella issue http://issues.jboss.org//browse/WFLY-266) and set their fix versions for both 8.2.0.CR1 and 9.0.0.Alpha1.

-- 
Jeff Mesnil
JBoss, a division of Red Hat
http://jmesnil.net/