[ https://jira.jboss.org/jira/browse/DNA-467?page=com.atlassian.jira.plugin... ] Randall Hauch reassigned DNA-467: --------------------------------- Assignee: Randall Hauch (was: Johnny Verhaeg) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/DNA-467?page=com.atlassian.jira.plugin... ] Randall Hauch updated DNA-467: ------------------------------ Fix Version/s: 0.7 (was: 0.6) Was targeted to fix in 0.6, but due to changes in the scope for 0.6 (and an interim release), this issue was moved into the 0.7 release. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/DNA-467?page=com.atlassian.jira.plugin... ] Randall Hauch commented on DNA-467: ----------------------------------- Actually, the work is pretty far along, and I'm actively working on it (now that 0.6 is out the door). I'm actually working on a (local) branch, and am hoping to commit what I have to trunk relatively soon. The design is based on the approach that JCR 2.0 has taken: multiple query languages are parsed into a single Abstract Syntax Tree (a "query model"), which is then executed by single engine. This means that 'dna-graph' will have a QueryEngine component parses string queries (given the name of the language, and a registered parser for that language that transforms the string query into an AST representation), plans and optimizes the query, constructs a processing plan, and delegates to a 'QueryProcessor' component to execute the processing plan and generate the results. These results are then returned to the caller of the QueryEngine. This design allows us to do a couple of different things. First, we can create multiple language parsers, yet have one engine. Second, the planning and optimization components can either be customized or replaced, so the whole query engine framework can be reused. Third, the query engine is completely independent of the execution, though the capabilities of the QueryProcessor can be taken into account during planning/optimization. We're actually providing a QueryProceessor implementation that uses Lucene indexes, and with this a way to populate the Lucene indexes by scanning the graph content and/or updating the indexes based upon events. We'll actually plug this query engine into the DNA JCR system in a couple of ways. Each connector will optionally be able to support querying its content (either by pushing down the queries to the underlying system, or by having a Lucene index and just instantiating the query engine within the connector). The federation connector will support queries by pushing down queries to the connectors being federated, and it will be able to manage a Lucene index for each connector that doesn't support queries. Then, since the JCR implementation will basically just implemented on top of the federated connector, it just has to delegate the queries down to the federated connector. Currently the following components are pretty complete (and have a lot of unit tests, though the tests are probably not sufficient): the QueryEngine, the AST classes, the query planner, the rule-based optimizer, and the QueryProcessor framework. The Lucene-based QueryProcessor implementation is at least 50% done and what remains is mostly integration. I'll be done with the JCR SQL2 parser (actually, the parser is complete and I have many unit tests, but only 50% of the parser is tested) in another day or so. What's left is: - commit the current state (I've been waiting to make sure all the pieces look like they'll fit together, and I'm probably no more than a few days away) - write an XPath parser - start integration testing of the query engine with the Lucene-based processor - add query support to the connector API - change some of our connectors to support queries (mostly just integrating the query engine and Lucene processor), though the federation connector will require some extra customization - add support for text extraction - expose through our JCR implementation - make sure the events work and are wired correctly, as well as set up mechanisms to manage the scanning and re-indexing of all content Another approach is to not yet put add query support to the connector API, but instead to just have our JCR implementation own the QueryEngine and Lucene-based processor. I'm not sure if this will save much time or effort (mostly because of federation), but it is an option. This is still a fair amount of work, but as you can see I think this design will be pretty powerful and will allow the connectors to really control how they handle queries. When it's committed to trunk, we'll be able to divvy up the tasks. WDYT so far? Yeah, this is a really rough description, but I do hope to document this more and blog about some of the details. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/DNA-467?page=com.atlassian.jira.plugin... ] Randall Hauch updated DNA-467: ------------------------------ Priority: Blocker (was: Critical) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/DNA-467?page=com.atlassian.jira.plugin... ] Randall Hauch updated DNA-467: ------------------------------ Complexity: High (was: Medium) -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/DNA-467?page=com.atlassian.jira.plugin... ] Randall Hauch commented on DNA-467: ----------------------------------- Changed how the Queryable and QueryEngine use Schemata. Before, the schemata instance was passed into the engine, and used for all queries. But this meant that the schemata instance could change the Table objects it returns. This would be more difficult to implement than if a different Schemata were passed with each query, since that Schemata instance can be a reflection of the tables at the time the query is implemented. And, Schemata can then be made to be immutable, not only simplifying the implementation but also ensuring that the schema information doesn't change during the processing of a query. Continued implementation of the dna-search components. Current status is that the SearchEngine is nearly complete (feature-wise), but has had little testing. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira

[ https://jira.jboss.org/jira/browse/DNA-467?page=com.atlassian.jira.plugin... ] Randall Hauch commented on DNA-467: ----------------------------------- Restructured the SearchEngine functionality to simplify it, and to change over to have implementations be extensions. To specialize, an implementation now simply subclasses the SearchEngine, and this is instantiated directly. The SearchEngine is also now oriented around a RequestProcessor. It's possible to obtain a RequestProcessor to issue multiple requests within a single connection, and this will make it significantly easier to reuse within a connector implementation (as well as within the 'dna-repository' module). Also, the SearchEngine's abstract RequestProcessor implementation provides default implementations for crawling and indexing entire subgraphs to generate the required UpdatePropertiesRequest and CreateNodeRequest stream. Specializations are responsible for fully-implementing the RequestProcessor to handle all the different kinds of requests, including AccessQueryRequest and FullTextSearchRequest. Thus, the LuceneSearchEngine can be instantiated directly, as it is now a subclass of SearchEngine. It also exists in a new 'extensions/dna-search-lucene' project, and continues to use the two-index design that was implemented previously. This new implementation uses a customized RequestProcessor implementation. This also should make it easier to process Changes via the request processor's methods. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://jira.jboss.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira