[dna-issues] [JBoss JIRA] Commented: (DNA-403) Federation connector does not behave properly in many situations

[ https://jira.jboss.org/jira/browse/DNA-403?page=com.atlassian.jira.plugin... ] Randall Hauch commented on DNA-403: ----------------------------------- Implemented a new federation connector (in 'dna-graph') that addresses most if not all of the concerns listed in the issue. Some changes were made to the requests (e.g., addition of DeleteChildrenRequest) and ExecutionContext and Location. However, most of the changes were in the addition of the 'org.jboss.dna.graph.connector.federation' package and its mostly package-level classes. The new federation connector does not use any caching. This was done in the interest of keeping the connector simple (or as simple as it can be), and if caching becomes useful then it can be added back in. The new design also uses several queues and an ExecutorService (owned by the FederatedRepositorySource instance) so that processing of the incoming requests are pipelined and processed by multiple threads. (In very simple situations, the processing is done in one thread.) The main thread spawns off a thread to process the incoming requests and "fork" these requests by determining how each request should be projected onto the underlying sources and then submitting these projected requests into the source connector's queue (where they are worked asynchronously by the same ExecutorService). The main thread, after starting the forking thread, then begins to process the federated requests, blocking until the corresponding source requests are completed, and then joining the results and applying to the original federated request. There are several major benefits of this approach. The first is that all requests submitted to an underlying source are processed in a single atomic transaction that corresponds to the federated connector's transaction. This means that canceling a single request will result in all other underlying connectors canceling their operations. It also means that integration with JTS/JTA will be much easier, as the commit/rollback activities will correspond directly with the federated connector's activities. (Source enlistment would be done only when a source is needed because its content is involved in at least one of the federated requests.) Secondly, this fork/join approach allows each connector to process its requests independently of the others and in separate threads. Thus, a single federated request that projects into multiple source requests will make maximum use of the available CPU (via the ExecutorService). It also means that this will shorten the processing latency (that is, the time to process the federated requests as seen by the client). Configuring this new federated connector uses the same configuration repository structure, albeit with a few minor changes. The first is that the caching information is not used. The second is that each projection can now be optionally marked as "read only" (via a "dna:readOnly" boolean property on the projection), which can improve performance by reducing the number of potential projections used during update requests. There still are a number of limitations regarding updates. The federation connector is only able to process create and updates if the federated request unambiguously applies to exactly one node in one projection. Copy and move requests only work when the 'from' and 'into' nodes each map to exactly one node in one projection AND where those nodes exist in the same source. (So, for example, 'from' and 'into' need to map to one node each, but those nodes may be in separate projections or separate workspaces as long as those nodes are in the same source.) Delete requests are applied to all appropriate writable projections. At this point, the old federation connector still exists in its own extension projection. However, as the examples are changed to use this new one, the old connector project will be removed. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira