[wildfly-dev] Proposal to add notifications to WildFly management model and API

[Jeff Mesnil]

# Add Notification support to WildFly Management

Tracked by https://issues.jboss.org/browse/WFLY-266

Use Cases
---------

Notifications are an useful mechanism to observe management changes on WildFly servers.
It allows an administrator to be informed of changes outside of his own actions (e.g. a server has been killed, a new application is deployed, etc.)

Currently WildFly lacks notifications and users that were depending on JMX notifications in previous versions have no similar feature to use.

The most expected use cases for WildFly notifications are:
- enhance UX for Web console. Using notifications, the Web console could notify the users of changes outside its own actions.
- replacement for JMX notifications. Users that were listening for JMX notifications to observe management changes would have a similar feature using WildFly own notifications
- integration with JMX. Notifications emitted by WildFly could be converted and made available using JMX notifications (including notifications for mbean registered/unregistered)

Part 1: Notification Definition
-------------------------------

A resource will define the notifications it emits. These definitions will be added to the attributes and operations definitions on a resource.

      {  
	  "description" => "A manageable resource",  
          "attributes" => {  
	     ...  
          },  
	  "operations" => {  
	     ...  
          },  
	  "notifications" => {  
	     "resource-added" => {
	        ...   
	     }  
	  },  
	  "children" => {  
	     ...  
          }  
       }  

The description of a notification will be composed of:

* type - String - the type of notification (resource-added, server-stopped, etc.)
* description - String - i18ned description of the notification
* access-constraints - the RBAC access constraints that controls who can receive the notifications
* data-type - ModelType or complex structure - optional - only present if the notification will have a data value. data-type will detail the structure of the data value, enumerating the value's fields and the type of their value

The read-resource-description will be enhanced with a notifications parameter (boolean) to include the notifications descriptions (default value is false, same as the operations parameter).

The ManagementResourceRegistration interface will be enhanced to register a notification definition with registerNotification(NotificationDefinition notification). The NotificationDefinition interface corresponds to the detyped representation of a notifications and comes with a builder API.

Part 2: Emitting a notification
-------------------------------

A notification can be emitted in any OperationStepHandler using the OperationContext.emit(Notification method)

    public void execute(OperationContext context, ModelNode operation) throws OperationFailedException {

       // perform some actions
       ...
    
       context.emit(new Notification(SERVER_RESTARTED_NOTIFICATION, address, ROOT_LOGGER.serverHasBeenRestarted()));
       context.stepCompleted();
    }

The notification is *not* emitted (i.e. delivered to interested parties) when OperationContext.emit() is called. It is emitted at the end of the operation step only if it is successful. A call to OperationContext.emit() will have no effect if the operation is rolled back.
Notification emission is done asynchronously using the server thread pool and does not block the execution of the operation that triggered the notification: having zero or any notification handlers must have no impact of the execution of the operation.

A Notification is a simple Java class that represents the notification. It is composed of:

* type - String - the notification type
* address - PathAddress - the address of the resource that emits the notification
* message - String - the i18ned description of the message
* timestamp - long - the timestamp of the notification. It is set when the Notification object is created.
* data - ModelNode - optional - a detyped representation of data associated to the notification. If a notification includes a data field, its definition must describe it (in its data-type parameter).

If RBAC is enabled, the notification access-constraints will be checked to ensure that the handler have the required privileges to receive the notification. Notification will potentially contain critical information (e.g. if a security-credential attribute is updated, the notification will contain its old and new values) and must be constrained accordingly.

Part 3: Global Resource Notifications
——————————————————

In the same way that some operations are available for any resource (e.g. add, remove, read-resource-description), some notifications will be added to any resource of WildFly management model:

* resource-added - when a resource is added, it emits a resource-added notification
* resource-removed - when a resource is removed, it emits a resource-removed notification
* attribute-value-written - when a write-attribute operation is performed successfully on a resource, it emits a attribute-value-written    notification. The notification's data field contains the following information:
  * name - String - the name of the attribute
  * old-value - the detyped representation of the previous value of the attribute
  * new-value - the detyped representation of the new value


Part 4: Notification Handlers
——————————————

Any interested parties can receive notifications by registering a NotificationHandler using the ModelController.getNotificationSupport().registerNotificationHandler(source, handler, filter) method.

The source is a path address to handle notifications emitted by resources at this address. 
The NotificationHandler is an interface with a single handleNotification(Notification notification) method.
The isNotificationEnabled(Notification notification) is an interface with a single isNotificationEnabled(Notification notification) method to filter out uninteresting notifications.

There is a similar unregister method to unregister a (handler, filter)

To be useful, the source path address will have to accept wildcards for the address' values:
* /subsystem=messaging/hornetq-server=* to receive notifications emitted by any hornetq-server resources
* /subsystem=messaging/hornetq-server=*/jms-queue=* to receive notifications emitted by any jms-queue on any hornetq-server resources

Wildcards for address' keys or key/value paris are not allowed (/subsystem=messaging/*=*/jms-queue=* and /subsystem=messaging/*/jms-queue=* are not valid).

This notion of wildcard for the resource addresses should be made to match current usage (e.g. in the CLI).

The main reason for the wildcard is for the resource-added/resource-removed notifications. I find more intuitive to have the notifications at the same resource-level than their corresponding add/remove operations. However until the resource is created, there is no way to register a notification listener on it without using a wildcard.
If that proves problematic, we could change this approach with two alternatives:
* have a single well-known resource emit the notifications for all resource (that's the JMX approach). A likely candidate would be /core-service=management
* the resource-added/-removed notifications can be emitted by the resource parents (but it only fixes the issue for the last leaf of the address tree…)

I still have questions about RBAC enforcements and it is possible that the registration of a handler will have to be done with additional metadata identifying the user roles wrt RBAC...

Part 5: Domain Notifications
——————————————

Notifications are also intended to work in domain mode. In particular, they will be used to observe server state.

The following notifications will be emitted by resources at /host=XXX/server-config=YYY (i.e. the resource to start/stop/etc. a server):
* server-started
* server-stopped
* server-restarted
* server-destroyed
* server-killed

Part 6: Integration with local JMX 
—————————————————

The jmx subsystem will be updated to leverage the WildFly notifications and expose them as MBean notifications in our jmx facade for the management model:
* the WildFly notification description will be converted to MBeanNotificationInfo and added to the MBeanInfo
* when a JMX notification listener is added to an ObjectName, a WildFly NotificationHandler will be added to the path address corresponding to the ObjectName.
* depending on the user feedback, we may provide a hack to convert some WildFly notifications to their well-known JMX equivalent notifications (e.g. resource-added => jmx.mbean.registered).

In a first step, integration will be limited to use of JMX locally. Remoting will not be supported.

Part 7: Integration with Remote Management API
———————————————————————

We will enhance the remote management native API to register/unregister a notification handler from the ModelControllerClient

    void registerNotificationHandler(ModelNode resourceAddress, NotificationClientHandler handler, NotificationClientFilter filter);

The client contract will have to taken into account reconnection when server is reloaded (possibly by caching the handler & filter and register them again after reconnection to the server...)

The Management HTTP API will also be enhance to support notifications with its REST API.
A neat addition will be to provide a browser-specific way to push notifications to the browser (e.g. using Server-Sent Events or Web Sockets).
=> the Web Console is the recipient for this feature and will have their say in how they prefer to consume notifications

Part 8: Integration with Remote JMX
—————————————————

Once the WildFly Management API will support notifications (for both native and HTTP), we can add support to JMX remotely (if there is any user interest for it).

Part 9: Web Console UX improvement
—————————————————

Once the Management HTTP API supports notifications, the Web console can leverage it to improve its UX.

This is a task that touch different parts of the app server (mainly in wildfly-core though) and I intend to split it in different JIRA issues (approx. one for each part) that can be merged one after the other instead of a big huge commit.

What do you think?

jeff


-- 
Jeff Mesnil
JBoss, a division of Red Hat
http://jmesnil.net/