[
https://jira.jboss.org/jira/browse/DNA-403?page=com.atlassian.jira.plugin...
]
Randall Hauch commented on DNA-403:
-----------------------------------
Implemented a new federation connector (in 'dna-graph') that addresses most if not
all of the concerns listed in the issue. Some changes were made to the requests (e.g.,
addition of DeleteChildrenRequest) and ExecutionContext and Location. However, most of the
changes were in the addition of the 'org.jboss.dna.graph.connector.federation'
package and its mostly package-level classes.
The new federation connector does not use any caching. This was done in the interest of
keeping the connector simple (or as simple as it can be), and if caching becomes useful
then it can be added back in. The new design also uses several queues and an
ExecutorService (owned by the FederatedRepositorySource instance) so that processing of
the incoming requests are pipelined and processed by multiple threads. (In very simple
situations, the processing is done in one thread.) The main thread spawns off a thread to
process the incoming requests and "fork" these requests by determining how each
request should be projected onto the underlying sources and then submitting these
projected requests into the source connector's queue (where they are worked
asynchronously by the same ExecutorService). The main thread, after starting the forking
thread, then begins to process the federated requests, blocking until the corresponding
source requests are completed, and then joining the results and applying to the original
federated request.
There are several major benefits of this approach. The first is that all requests
submitted to an underlying source are processed in a single atomic transaction that
corresponds to the federated connector's transaction. This means that canceling a
single request will result in all other underlying connectors canceling their operations.
It also means that integration with JTS/JTA will be much easier, as the commit/rollback
activities will correspond directly with the federated connector's activities.
(Source enlistment would be done only when a source is needed because its content is
involved in at least one of the federated requests.)
Secondly, this fork/join approach allows each connector to process its requests
independently of the others and in separate threads. Thus, a single federated request
that projects into multiple source requests will make maximum use of the available CPU
(via the ExecutorService). It also means that this will shorten the processing latency
(that is, the time to process the federated requests as seen by the client).
Configuring this new federated connector uses the same configuration repository structure,
albeit with a few minor changes. The first is that the caching information is not used.
The second is that each projection can now be optionally marked as "read only"
(via a "dna:readOnly" boolean property on the projection), which can improve
performance by reducing the number of potential projections used during update requests.
There still are a number of limitations regarding updates. The federation connector is
only able to process create and updates if the federated request unambiguously applies to
exactly one node in one projection. Copy and move requests only work when the
'from' and 'into' nodes each map to exactly one node in one projection AND
where those nodes exist in the same source. (So, for example, 'from' and
'into' need to map to one node each, but those nodes may be in separate
projections or separate workspaces as long as those nodes are in the same source.) Delete
requests are applied to all appropriate writable projections.
At this point, the old federation connector still exists in its own extension projection.
However, as the examples are changed to use this new one, the old connector project will
be removed.
Federation connector does not behave properly in many situations
----------------------------------------------------------------
Key: DNA-403
URL:
https://jira.jboss.org/jira/browse/DNA-403
Project: DNA
Issue Type: Bug
Components: Connectors, Federation
Affects Versions: 0.4
Reporter: Randall Hauch
Assignee: Randall Hauch
Priority: Blocker
Fix For: 0.5
The federation connector is not well behaved when it comes to a number of very common
situations. In particular, there seem to be a lot of failures when attempting to
repeatedly find nodes by UUID. I suspect that the cache layer is not being updated
correctly.
There are also several issues with the fundamental design. Currently, all operations
against the cache and any sources are performed serially, despite the fact that each
connector's processing of requests submitted to it is complete independent of all
other connectors. This means that all source connector processing could be parallelized.
Another issue is that when a CompositeRequest is submitted to the federation connector,
these requests are treated as if they are being done within a single (logical)
transaction. However, often times the processing of a request submitted to the federated
connector results in one or more requests being made to the source connectors, and these
source requests are submitted individually (rather than in a single CompositeRequest
corresponding to that submitted to the federated connector). Thus, there is no tie
between the source connections and federated connection. Although we don't supported
distributed transactions, this certainly doesn't come close to helping align the
logical transactions of each connector.
Third, the federation connector does not optimize for some common situations, such as a
single mirror projection (e.g., "/ => /", where the federation connector is
really just mirroring another source), an offset projection (e.g., "/ =>
/a/b" or "/c/d => /", where the federation connector is mirroring a
subset of another connector or prepends a leading path), a mirror-with-branch case (e.g.,
"/ => /" for source A, and "/jcr:system => /jcr:system" for
source B). The latter condition is likely to be used in the JCR implementation to project
a single "/jcr:system" branch (which will exist in its own workspace in a JCR
repository) into each normal JCR workspace.
Finally, as just mentioned, the federation connector has the potential to be used in a
number of situations within other non-extension projects (e.g., "dna-jcr").
This is not really feasible with the current connector, since it is a normal extension and
not a first-class feature of the 'dna-graph'. Therefore, the federation connector
should be moved into 'dna-graph' similar to the In-Memory connector.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://jira.jboss.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira